Plotting live data using Highcharts and a REST API

Highcharts is a JavaScript charting framework, similar to D3.js, plotly.js and Google Charts. It enables the creation of various types of interactive charts which can easily be integrated on a web site.

The King’s College London API provides live air quality data for sites across London. This REST API exposes data from the database in either JSON or XML. Calling the API returns data in JSON format (as opposed to HTML), allowing the data to be directly used in Python. The following chart was created using this API together with HighCharts and Flask.

Flask is used since HighCharts is written in HTML5/JavaScript and therefore requires a web browser.   The code for this web app is contained within this GitHub repository:  https://github.com/paulos84/airapp3

Within the charts.py file in the views directory, the get_json function returns a dictionary of air quality monitoring data requested from the London Air API. The function takes in values which specify the site and number of previous days data the user is interested in. String formatting is then used to generate the desired endpoint as a string which is passed to the requests get method.

Before the requests library was released, sending HTTP requests relied upon the verbose and cumbersome urllib2 library. The requests library greatly reduces the lines of code needed and is well suited to making RESTful API calls. The get method requires a URL as an argument and allows you to pass optional parameters such as http request headers (e.g. login credentials). Requests built-in JSON decoder, called by request.json(), converts the JSON response into a Python dictionary, which in this case contains many layers of nesting.

The get_data function uses list comprehensions to create lists of pollutant values and the hours (for the x and y axes). To avoid any KeyErrors, empty strings are returned instead of None for missing data points. The get_data function passes a dictionary of these lists to the make_chart function which has a decorator specifying the url. By providing ‘detail.html’ as a positional argument, Flasks render_template method passes the key-values pairs required by HighCharts in order create the desired chart. This html template containing the HighCharts JavaScript code is contained within the templates directory.

Advertisements

Adjusting Mouse Scroll Speed in Mac OS Sierra

For many people, the default range of mouse scrolling speeds offered in the Mac OS System Preferences tab does not offer fast enough tracking speeds, especially on larger screens. In previous releases of Mac OS X, 3rd party preference tabs could be used to allow faster speeds and other formsof customisation (including mouse acceleration). Unfortunately, with the release of Mac Os Sierra, several of the internal calls used by these 3rd party utilities were removed, disabling their functionality.

There is still a way to enable faster mouse scroll speeds in Mac OS Sierra though, as detailed in this Apple Support article.

The new process is as follows:

  1. From the Apple menu, choose System Preferences.
  2. From the System Preferences window, select Accessibility.
  3. In the left sidebar, select Mouse & Trackpad.
    System Preferences
  4. Click the Mouse Options button.
  5. In the sheet that appears, use the slider to adjust the mouse scrolling speed, including acceleration.
    Scrolling Speed in System Preferences

 

The “Mouse & Trackpad” accessability panel sheet gives quite a few good options, although still not as many as some of the 3rd party preference utilitities did.

Dictionaries in Python

Python’s way of storing key-value pairs, a fundamental data structure in computer science. The data type is summarized in the official documentation as “an unordered set of key: value pairs, with the requirement that the keys are unique”. Dictionaries can be indexed by any immutable data type and the stored values accessed in the following ways:

value = d.get[key]

value = d.get(key)

value = d.get(key, "no data")

Whereas using [key] will return a KeyError if the key does not exist, the .get method will either return None, or a default value if specified as an optional parameter. Values within nested dictionaries, such as deserialized JSON data, can be accessed by the successive use of [key] or .get(key):

sales = {'data':{'orders':{'january':240}}}

sales['data']['orders']['january']

sales.get('data').get('orders').get('january')

sales['data']['orders'].get('january')

The following are all valid ways of creating dictionaries:

my_dict = {'key1': 'value1', 'key2': 'value2'}

my_dict = dict(key1='value1',key2='value2')

my_dict = {x: x**2 for x in values}

my_dict = dict(zip(keys, values))

When the keys are simple strings, it can be useful to pass in the keys as keywords to the dict() constructor. This is the most performant way of creating dictionaries and useful for the generation of arbitrary keys and values. Using the zip function inside the dict() constructor is particularly useful for creating dictionaries from lists of keys and values.

Dictionaries are unordered, except in Python 3.6+. To store the insertion order of keys, the dictionary sub-class OrderedDict can be used after importing it from the collections module in the standard library.

Data visualization libraries for Python

Matplotlib and pandas (a library built on top of NumPy) are a powerful combination for processing and plotting data. The default plotting styles of matplotlib are somewhat basic, but with recent versions the aesthetics can be improved using the style sub-package. A list of available styles can be obtained using the style.available attribute:

from matplotlib import pyplot as plt, style
>>> print (plt.style.available)
['seaborn-deep', 'seaborn-dark', 'fivethirtyeight', 'dark_background', 'seaborn-colorblind', 'seaborn-bright', 'seaborn-notebook', 'seaborn-whitegrid', 'seaborn-dark-palette', 'seaborn-ticks', 'seaborn-pastel', 'seaborn-poster', 'classic', 'seaborn-white', 'grayscale', 'seaborn-paper', 'seaborn-muted', 'seaborn-talk', 'ggplot', 'seaborn-darkgrid', 'bmh']

Then just call style.use() within the code used to generate a plot:

plt.style.use(‘seaborn-white’)

Seaborn is a library built on top of matplotlib. It provides various useful plotting functions and the plots it produces tend to be visually attractive. Seaborn is especially useful for exploring statistical data and for use with more complex data sets.

The choice of library should largely depend upon the desired visualization. Matplotlib on its own is very powerful and should be used for simple bar, line, pie, scatter plots etc. More complicated plots will require significantly more lines of code and seaborn will usually be more appropriate in these cases.

Bokeh was created with the aim of providing attractive and interactive plots in the style of the JavaScript D3.js library. Since Bokeh is higher level than D3.js, interactive visualizations can generally be created with much less effort. The documentation is fairly comprehensive, however the library is still under heavy development so may best be avoided if future compatibility is a potential issue.

Avoiding multi-table inheritance in Django Models

Model inheritance does not have a natural translation to relational database architecture and so models in Django should be designed in order to avoid impact on database performance. When there is no need for the base model to be translated into a table abstract inheritance should be used instead of multi-table inheritance.

Given the following model:

class Person(Model):
  name = CharField()
  …

class Employee(Person):
  department = CharField()
  …

Two tables will be created and what looks like a simple query to the Employee child class will actually involve a join automatically being created. The same example with abstract = True in the Meta class allows abstract inheritance:

class Person(Model):
  name = CharField()
  …

class Meta:
  abstract = True

class Employee(Person):
  department = CharField()
  …

By putting abstract = True, the extra table for the base model is not created and the fields within the base model are automatically created for each child model. This avoids unnecessary joins being created to access those fields. This way of using model inheritance also avoids repetition of code within the child classes.

How to Install Apache Tomcat on Mac OS Sierra

Our previous tutorial on installing Tomcat on El Capitan had a lot of interest, so here it is an updated (but broadly similar) tutorial for MacOS Sierra.

In this tutorial, we will be using the open-source package manager Homebrew. If you’re not already using homebrew, check out its popularity on GitHub. It’s highly recommended for use with developing on Macs as it makes keeping track of installed software 100 times cleaner than doing it manually (everything is stored in one place, packages are easy to remove, upgrade and find configs for).If you don’t already have Homebrew, install it with:

First, open your Mac terminal window. If you don’t already have Homebrew, install it with:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"


Step 1: Install Tomcat

Now we can easily install and track the version of Tomcat we’re using and config files etc with the following command:

brew install tomcat

This will take care of the downloading, installation and configuration of Tomcat and manage its dependencies as well. Take note of the output, brew commands are typically really good at displaying concise but useful info, errors and other help.

Homebrew keeps all installed packages (called “kegs”) in a “Cellar” folder – they like their beer references. By default, this is located in the /usr/local/ directory, so there should now be a tomcat folder listed by the command:

ls /usr/local/Cellar

Although an easier shortcut is available as:

brew list

To get more info on Tomcat specifically:

brew list tomcat

This should output something similar to:

$ brew list tomcat
/usr/local/Cellar/tomcat/8.5.15/bin/catalina
/usr/local/Cellar/tomcat/8.5.15/libexec/bin/ (15 files)
/usr/local/Cellar/tomcat/8.5.15/libexec/conf/ (10 files)
/usr/local/Cellar/tomcat/8.5.15/libexec/lib/ (25 files)
/usr/local/Cellar/tomcat/8.5.15/libexec/temp/safeToDelete.tmp
/usr/local/Cellar/tomcat/8.5.15/libexec/webapps/ (573 files)
/usr/local/Cellar/tomcat/8.5.15/RELEASE-NOTES
/usr/local/Cellar/tomcat/8.5.15/RUNNING.txt

The listed binary file Catalina is the “servlet container” used to run the Apache Tomcat server. This can now be started using the command:

catalina run &

The ampersand at the end makes the process run in the background, so after pressing the return key you get your terminal back but Catalina is still running in the background, you can remove it if you want to keep a dedicated window on the process.

Catalina can be stopped using:

catalina stop

 

Step 2: Configure Tomcat

Apache Tomcat comes with an inbuilt GUI management suite, however, for security reasons, this is disabled by default (to avoid dangerous default usernames/passwords). To enable the GUI manager, first edit the file:

nano /usr/local/Cellar/tomcat/8.5.15/libexec/conf/tomcat-users.xml
  • Nano is a text editor that comes with MacOS, although any will do.
  • If the file is blank or not found, you probably have a slightly different version of Tomcat installed, hit ctrl + x to exit, then run the command brew list tomcat and replace the above version number with the one you have installed.

Scroll to the bottom of the file and you should see several user entries surrounded such as:

<!–
<role rolename=”tomcat”/>
<role rolename=”role1″/>
<user username=”tomcat” password=”<must-be-changed>” roles=”tomcat”/>
<user username=”both” password=”<must-be-changed>” roles=”tomcat,role1″/>
<user username=”role1″ password=”<must-be-changed>” roles=”role1″/>
–>
</tomcat-users>

The <!– …  –> means these users are commented out. Leave them as they are and add a new entry after but above the </tomcat-users> line with the “manager-gui” role:

<user username=”someUser” password=”somePassword” roles=”manager-gui”/>

So the file now looks like:

<!–
<role rolename=”tomcat”/>
<role rolename=”role1″/>
<user username=”tomcat” password=”<must-be-changed>” roles=”tomcat”/>
<user username=”both” password=”<must-be-changed>” roles=”tomcat,role1″/>
<user username=”role1″ password=”<must-be-changed>” roles=”role1″/>
–>
<user username=”someUser” password=”somePassword” roles=”manager-gui”/>
</tomcat-users>

Obviously, you should use a unique username & password for security!

Now, start Catalina again:

catalina run
  • If you get errors, you probably need to stop it first, use catalina stop
  • By default, Tomcat runs on port 8080. There’s a really useful command to see what services are running on this a port: lsof -i :8080 (you may need to prefix with sudo for admin protected ports like 80).

Now if you go to the following page you should see a management GUI:

http://localhost:8080/manager/html

Here you can deploy .war files or exploded directories using the Deploy console and existing servlets are listed. You can even try visiting the already deployed servlets by appending the listed path to localhost:8080 (the default tomcat port):

So, the following should give the docs & some examples:

http://localhost:8080/docs
http://localhost:8080/examples

Enter the .war file location you wish to deploy and the desired path into the Deploy section and your application should now be listed as running with its path. Additionally, this can be used with an IDE like Netbeans or IntelliJ to run & debug servers using their configuration windows.

Any questions, please ask in the comments, and please share this article if you found it helpful.

PBCopy Mac command

A couple of useful Mac terminal (or iTerm) commands are the PBCopy & PBPaste. Here is a demonstration of how they work.

There are two basic ways to copy using pbcopy:

$ pbcopy < source_file.txt
copies text in source_file.txt

$ pwd | pbcopy
copies the output from pwd (the current directory) to the Mac clipboard

The copied text can now be pasted (either using menu’s or the CMD + V keyboard shortcut) into other applications in the usual way.

PBPaste works using the same syntax but in reverse:

$ pbpaste > destination_file.txt
pastes the current Mac clipboard to destination_file.txt

$ cd `pbpaste`
change directory to what's is currently stored in Mac clipboard