Jon Perry's Blog

Lambda Expressions in Python 3

In Python 3, Lambda Expressions are supported. Like Java 8, they are anonymous functions, which are very handy. They mix simplicity with readability and reusability. Their functional programming style makes them handy for writing conditional and mathematical expressions.

Java 8

list.forEach(a -> {

System.out.println(Math.sqrt(a));

}

Python on the other hand defines lambdas in a different way.

object = lambda identifier : identifier

object = lambda identifier : iif identifier == someVariable

As emphasised above, we need to specify the lambda keyword. We provide an identifier (this can be a String/List/Integer/Float/Double or even an object); this is our input variable. You can define multiple identifiers. It provides us the ability to define an expression or operation we want to perform on the identifier(s). The semi-colon (:) allows us to do this. object just denotes an object which contains our lambda function. We can call the Lambda like so:

object(45)

Here’s a simple and better implementation of a Lambda:

even = lambda num : num % 2
print(even(45)) # prints 1, which is odd

Try it online: https://repl.it/languages/python3

Common Python Libraries

In my day job at the BBC, we use Python. It’s a great scripting language; highly versatile, well supported and mixes simplicity with complexity, if need be.

One of the challenges of my day job to date, has been learning AWS, Ruby and Python as I go. This presents challenges as it’s not easy to adequately take in and absorb all the information thrown at you. Particularly when a language has its own ecosystem and SDK to learn.

Like Java, Python has its own standard modules. Here’s a few modules that are commonly used in most scripting applications.

import os

The OS module provides lots of very useful classes and methods that allow developers to interact with the operating system. For instance, we can execute bash commands using the os.system(“echo ‘hello'”) method to list, create, delete and modify directories. Alternatively, you can use the shutil module instead, which provides a higher level of abstraction from os.

import math

The math module is also very useful…

import re

The re module is a confusing name given its purpose. It is useful for pattern matching using regular expressions (Regex).

import urllib2

The urllib2 module is very useful for when you need to interact with the internet. It’s useful for creating HTTP clients and sending requests to external web services using REST.

from datetime import date

The datetime module…

import zlib

The zlib module is used for data compression. It supports data compression across .zip, .tar.gz, etc. Pretty cool.

Summary

import os
import shutil # higher abstraction from os
import math
import re
import urllib2
from datetime import date
import zlib

AWS Lambda: Cold Starts

At work, I’ve been working with Lambdas. They’re interesting and quite exciting things to work with. They’re simple and very effective. They’re not designed to compute and process intense volumes of data beyond periods after 15 minutes (Lambdas can be warmed up between a minimum of 5 to 15 minutes), but for a lot of tasks, they’re very well suited. They can scale very quickly through throttling and there is a very nifty 50MB which can be used in the /tmp directory.

The major drawback for Lambdas are cold starts – particularly when working with a JVM-based language. It’s one of the major headaches any software engineer has to deal with; basically ‘how does one reduce the time it takes to execute X, while the Linux-based VM starts-up?’. It’s something you have to think about, to improve throughput.

So when a Lambda hasn’t been warmed up, it’s in a cold state. Suppose a message is sent to a queue (SQS) or an item is being inserted into a DynamoDB table, a signal is sent to the container (a Lambda is virtualised), and the VM (OS) then starts up. AWS Lambda is based on Firecracker, which is a MicroVM, that’s written in Rust (it’s similar to C++, but designed for better memory utilisation and thread safety). A MicroVM is based on a stripped down distribution of the Linux Kernel VM. By design, it has a very low memory overhead: just 5MiB. It potentially allows for hundreds or thousands of containers to be run on a single machine, which is where throttling can come in.

Its optimised design helps the VM to start up quickly. Unfortunately the start time of the MicroVM is just the first stage and if not setup properly, the JVM can add more milliseconds than you’re willing to settle with. Using a custom JVM (Java 11) and one of the JIT (Just in Time) compilation arguments can alleviate this, but it obviously has its performance drawbacks.

So how can we optimise the JVM to reduce cold starts? Well a good place is to go back to basics of how the JVM works. We all know that objects are created and at compile time, they’re compiled into byte code, against the maximum supported byte code version on the JRE (Java Runtime Environment) in the JVM that it’s written against.

When that bytecode is executed, the JVM stores objects into heaps of memory and stores its reference in the stack. Since the JVM can be an expensive beast in terms of CPU/RAM requirements, understanding the relationship between CPU clock cycles and read/write executions to each block of memory is important.

A good strategy is to limit what objects are allocated memory blocks in the JVM’s heap and call object methods statically. This helps the garbage collector and reduces creation and destruction of objects. Obviously if you need an array of objects in your code, then they’ll need to create an instance and create a reference in the stack to it’s place in the heap. However for methods in an object that aren’t needed that often, using static methods can be very useful.

Another strategy is to design your Lambda function with simplicity and versatility in mind. Good Object-Oriented Design can be very helpful and it will allow you to understand the level of interaction between CPU clock cycles and read/write executions to each block of memory. Design it in steps and use the JDK as appropriately as possible when implementing your code! It will thank you; it can be indicated through writing your function’s code using Test Driven Development (TDD) and reducing the time it takes to execute each unit and integration test, through refactoring.

I hope this post has been useful and insightful. I’m still learning myself, so if I learn anything more or new I will update it!

https://aws.amazon.com/blogs/opensource/firecracker-open-source-secure-fast-microvm-serverless/

Python – Throwaway Variables

I’ve recently been using Python in my day job, and it’s a pretty neat language – Industrial Light and Magic, Google, Facebook, etc. all use it. It is one of the most versatile languages out there, along with Java.

A particular trick that I discovered recently, is throwaway variables. They are very useful for extracting values. They are denoted using an underscore and comma _, to tell the interpreter to ignore the next set of alphanumeric symbols.

Here’s an example:

names = "Peter, Daniel, Sarah, Tom, Danielle, Amy, Sebastien"
_, _, a, _, b, c, _ = names.split(",")
print "{}, {}, {}".format(a, b, c)

Output: Sarah, Danielle, Amy

Pretty neat!

Exception Handling in Java

Recently I have been learning how to use the exception handling types in Java’s JDK more appropriately.

Here are some golden rules to follow, when throwing an exception type. Much of how you use checked and unchecked exception types, depends on where you want to throw an exception in the application’s stack.

IllegalArgumentException

  • Used and useful when parsing an object or string that fails.
  • A good example could be validating whether a URL or email matches a given format.

NullPointerException

  • A difficult exception to throw compared to IllegalArgumentException.
  • It’s recommended you stay away from using them.
  • You do not want to mix bad NPEs with NPEs thrown intentionally.
  • It is recommended you throw IllegalArgumentException instead.
  • Just return null when encountering a value that has not been set.

RuntimeException

  • The foundational (base) exception class in Java.
  • It is commonly used to catch unchecked exceptions.
  • It is particularly useful to throw, when processing something at a higher level, for instance, parsing with an external properties file.
  • It is common to extend a custom exception class using the RuntimeException class, since it is considered best practice to do so, unless extending a super class with more unique exception handling capabilities.

Git Commands – Help Sheet

Essential Git commands for effective software development.

Undo a commit

git reset HEAD –hard

Pulls latest changes from the repo and rebases changes from the master branch

git pull origin remote-branch —rebase master

Resets stage (doesn’t undo changes)

git reset –soft HEAD

Resets stage (undo’s any uncommitted changes)

git reset –hard HEAD

Continues iterating through merged changes made from the rebased repo

git rebase –continue

Aborts merging rebased changes

git rebase –abort

Force pushes changes made to the local repo (usually made when a rebase has occurred when updating a branch)

git push origin branch –force

Fetches all remote branch names not in local

git fetch

Checks out a remote branch into local tracking that repo for changes

git checkout –track remote-branch

Create a new branch and switch to it

git checkout -b branch

Current local branch

git branch -l

List current, all local remote branches being tracked

git branch -a

Display URL to remote repository

git remote -v

Add New Repository URL

git remote add origin url

Change Existing Repository URL

git remote set-url origin repository-url

Delete a local branch

git branch -D branch

Delete a remote branch

git push origin –delete remote-branch

Stages a file for commit

git add path-to-file

Commits all staged files with commit message

git commit -am “message”

Stash Changes

git stash

List Stash Changes

git stash list

Apply Stash

git stash apply stash@{stashIndex}

Note: 0 is the most recent stash

Save Stash (with message)

git stash save “with message”

Delete Stash by Index

git stash drop stash@{stashIndex}

Delete All Stashes *

git stash clear

* This will delete ALL your stashed changes

Spring Web Flow – using Eclipse IDE (Tutorial)

I’ve been looking online lately for tutorials on Spring Web Flow, and I didn’t find any that were succinct, memorable, or very appealing. For every tutorial that you find online, that framework may be out of date, and another may be in date; but be horrendously long to read… This article helps challenge these attributes.

For this tutorial I am using:

IDE: Eclipse (Java EE)
Build tool: Maven
Framework: Spring v4 << the important bit ... and Java, obviously.

Step 1:

Create a new project in Eclipse. You’ll need to open ‘other’ option under the New menu heading. A new window will open – select Maven -> Maven Project.

Step 2:

Setup your Spring dependencies.

When you create a new Maven project (under File -> New -> Other…), you will need to use your default workspace location, using the ‘maven-archetype-webapp’ artifact ID. Give your Maven project a name, followed by the parent package that you want to create for this project.

Open the pom.xml file under the root of your application’s directory.

With your pom.xml configuration file open, you’ll need to specify to Maven what Java libraries your application will need to build, deploy and run.

To create a Spring Web Flow application, we’ll need three components belonging to the Spring framework, those being: Spring Core, Spring MVC and Spring Web Flow. We will also need Spring Boot, Spring Web, Spring Context and Spring Autoconfigure.

The framework itself isn’t one dependency, more-over is an assortment of components, (I presume) for the purpose of modularity and performance.




org.springframework.boot
spring-boot
1.2.3.RELEASE



org.springframework
spring-web
4.1.6.RELEASE



org.springframework
spring-core
4.1.6.RELEASE



org.springframework
spring-webmvc
4.1.6.RELEASE



org.springframework
spring-webflow
1.0.6



org.springframework
spring-context
4.1.6.RELEASE



org.springframework.boot
spring-boot-autoconfigure
1.2.3.RELEASE

Figure 1. Maven dependencies – feel free to copy n’ paste!

Step 2: Write code like MVC… not ABC…

We’ve got the important stuff out of the way. We can now focus on writing our Java code!

To begin with, we’re going to write the root of our application.

Start by creating a new Java class and name it something like Application. This will allow us to compile and execute the application on our local Tomcat server.

package com.learning.spring.webflow;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Application {

public static void main(String [] args) {
SpringApplication.run(Application.class, args);
}

}

This very small and simple class simply instructs the Spring boot component to execute our code against the JVM.

Next, we need to create a class which will act as the Controller (eventually) in our Spring Web Flow application.

// Spring MVC component
import org.springframework.stereotype.Controller;

// Spring Web component
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
public class HolidayBookingFormController {

@RequestMapping("/book-holiday-date")
public String holidayChooseDates() {

}

@RequestMapping("/book-holiday-guests")
public String holidayNumberOfGuests() {

}

@RequestMapping("/book-holiday-flights")
public String holidayChooseFlights() {

}

@RequestMapping("/book-holiday-extras")
public String holidayChooseExtras() {

}

@RequestMapping("/book-holiday-insurance")
public String holidayConfirmBooking() {

}

}

For each of the methods contained within this controller, each HTTP request maps (corresponds) with the JSP that will handle the presentation of our application.

Step 3: Create the JSP’s!

New Hobby…

So… we are 25 days into the New Year. It’s been ok, and despite the cold, things are going generally, ‘ok’.

For a long time, I’ve been exploring and developing experience in different disciplines encompassing Computer Science (mainly to get a job in software engineering), but I haven’t had time to explore a topic where I can unleash my creativity, never get bored technically, and generally enjoy a hobby with a lot of pleasure attached to it. That topic, is Computer Graphics!

I can’t wait to see how things develop. I’m going to make loads of mistakes, and probably adopt some bad practices (by mistake) at some stage, but I can’t wait to show the world what I do. I’ve been exposed to art, design and technical disciplines through my life and I really relish doing something a lot of Computer Science professionals never sadly explore in their spare time alongside a full time job.

Happy New Year!

It’s been quite a while since I last posted something on this blog. In that time I have finished my MSc and started a new job in the real world. In the spirit of the New Year and Resolutions, I have recently created my own custom theme for this blog (ooh er!).

The hope behind the change in the design is to put myself into a more focused, professional mindset, so I can use this little space on the inter-web to share my technical creations in my spare time, now that I am working in the commercial world of technology 🙂

Watch this space…

Dissertation Progress – Day 67

Tasks complete today:

  1. Coded intial monitoring system component – will implement Fuzzy Logic system tomorrow.
  2. Read paper ‘The Computer for the 21st Century’.