Plotting live data using Highcharts and a REST API

Highcharts is a JavaScript charting framework, similar to D3.js, plotly.js and Google Charts. It enables the creation of various types of interactive charts which can easily be integrated on a web site.

The King’s College London API provides live air quality data for sites across London. This REST API exposes data from the database in either JSON or XML. Calling the API returns data in JSON format (as opposed to HTML), allowing the data to be directly used in Python. The following chart was created using this API together with HighCharts and Flask.

Flask is used since HighCharts is written in HTML5/JavaScript and therefore requires a web browser.   The code for this web app is contained within this GitHub repository:

Within the file in the views directory, the get_json function returns a dictionary of air quality monitoring data requested from the London Air API. The function takes in values which specify the site and number of previous days data the user is interested in. String formatting is then used to generate the desired endpoint as a string which is passed to the requests get method.

Before the requests library was released, sending HTTP requests relied upon the verbose and cumbersome urllib2 library. The requests library greatly reduces the lines of code needed and is well suited to making RESTful API calls. The get method requires a URL as an argument and allows you to pass optional parameters such as http request headers (e.g. login credentials). Requests built-in JSON decoder, called by request.json(), converts the JSON response into a Python dictionary, which in this case contains many layers of nesting.

The get_data function uses list comprehensions to create lists of pollutant values and the hours (for the x and y axes). To avoid any KeyErrors, empty strings are returned instead of None for missing data points. The get_data function passes a dictionary of these lists to the make_chart function which has a decorator specifying the url. By providing ‘detail.html’ as a positional argument, Flasks render_template method passes the key-values pairs required by HighCharts in order create the desired chart. This html template containing the HighCharts JavaScript code is contained within the templates directory.


Switching databases in Symfony upon application events

We’re big fans of the PHP Symfony2 framework and Doctrine2 ORM but a recent project involved a REST API for a multi-client system where each client wanted their database fully separated from each other, but we only knew which client database to connect to once a user had logged in (so no easy config.yml and parameters.yml configuration).

The login gets handled via Java Web Tokens using the excellent Lexik JWT Authentication bundle. So the first call to /auth/login_check with a username and password retrieves a token along with a id of the client (added using the onAuthenticationSuccessResponse listener documented here. Subsequent requests supply a HTTP Authorization header with the token and a custom “client” header specifying the id of the client (also documented in the above link).

This gave us stateless authentication for our API, but we still had to actually switch the Doctrine dbal connection to point to the right database. The solution (inspired by this Stack Overflow question) involved an onKernelRequest() event listener to switch the database from a base database (holds login / authentication data and is referenced in the default connection settings – or whichever connection you’re overriding – of your config.yml file) to the desired database and update the tokenStorage object accordingly. As a tip though, it’s best to keep the base and individual databases schematically identical as possible for reasons explained later.

First create the Event Listener:

Then add this to the services.yml file:

Then add the client databases to the end of the parameters-dist.yml file:

The security.yml file looks something like:

This solution works fairly well in providing multi-database connectivity based on a site event (in this case user login) with the occasional bit of peering underneath’s Symfony’s hood. As mentioned above, keeping the base database schematically identical (i.e. same table structure, differing data) to your individual databases is very highly advisable. Some Symfony logic occurs before the onKernalRequest event gets fired (looking at your forms) so functionality limitations can otherwise occur there. Also parsers like the Nelmio Api Documentation bundle typically build pre-authentication (so whilst the base database is still selected).

Doctrine Migrations can also be enabled with a little tinkering (I may write a new post on how to do this if there is demand) but again, it helps if the schemas are identical (although isn’t vital).