Dictionaries in Python

Dictionaries are Pythons way of storing key-value pairs, a fundamental data structure in computer science. The data type is summarized in the official documentation as “an unordered set of key: value pairs, with the requirement that the keys are unique”. Dictionaries can be indexed by any immutable data type and the stored values accessed in the following ways:

value = d.get[key]

value = d.get(key)

value = d.get(key, "no data")

Whereas using [key] will return a KeyError if the key does not exist, the .get method will either return None, or a default value if specified as an optional parameter. Values within nested dictionaries, such as deserialized JSON data, can be accessed by the successive use of [key] or .get(key):

sales = {'data':{'orders':{'january':240}}}

sales['data']['orders']['january']

sales.get('data').get('orders').get('january')

sales['data']['orders'].get('january')

The following are all valid ways of creating dictionaries:

my_dict = {'key1': 'value1', 'key2': 'value2'}

my_dict = dict(key1='value1',key2='value2')

my_dict = {x: x**2 for x in values}

my_dict = dict(zip(keys, values))

When the keys are simple strings, it can be useful to pass in the keys as keywords to the dict() constructor. Dictionary comprehensions are the most performant way of creating dictionaries and are useful for the generation of arbitrary keys and values. Using the zip function inside the dict() constructor is particularly useful for creating dictionaries from lists of keys and values.

Dictionaries are unordered, except in Python 3.6+. To store the insertion order of keys, the dictionary sub-class OrderedDict can be used after importing it from the collections module in the standard library.

Advertisements

Data visualization libraries for Python

Matplotlib and pandas (a library built on top of NumPy) are a powerful combination for processing and plotting data. The default plotting styles of matplotlib are somewhat basic, but with recent versions the aesthetics can be improved using the style sub-package. A list of available styles can be obtained using the style.available attribute:

from matplotlib import pyplot as plt, style
>>> print (plt.style.available)
['seaborn-deep', 'seaborn-dark', 'fivethirtyeight', 'dark_background', 'seaborn-colorblind', 'seaborn-bright', 'seaborn-notebook', 'seaborn-whitegrid', 'seaborn-dark-palette', 'seaborn-ticks', 'seaborn-pastel', 'seaborn-poster', 'classic', 'seaborn-white', 'grayscale', 'seaborn-paper', 'seaborn-muted', 'seaborn-talk', 'ggplot', 'seaborn-darkgrid', 'bmh']

Then just call style.use() within the code used to generate a plot:

plt.style.use(‘seaborn-white’)

Seaborn is a library built on top of matplotlib. It provides various useful plotting functions and the plots it produces tend to be visually attractive. Seaborn is especially useful for exploring statistical data and for use with more complex data sets.

The choice of library should largely depend upon the desired visualization. Matplotlib on its own is very powerful and should be used for simple bar, line, pie, scatter plots etc. More complicated plots will require significantly more lines of code and seaborn will usually be more appropriate in these cases.

Bokeh was created with the aim of providing attractive and interactive plots in the style of the JavaScript D3.js library. Since Bokeh is higher level than D3.js, interactive visualizations can generally be created with much less effort. The documentation is fairly comprehensive, however the library is still under heavy development so may best be avoided if future compatibility is a potential issue.

Avoiding multi-table inheritance in Django Models

Model inheritance does not have a natural translation to relational database architecture and so models in Django should be designed in order to avoid impact on database performance. When there is no need for the base model to be translated into a table abstract inheritance should be used instead of multi-table inheritance.

Given the following model:

class Person(Model):
  name = CharField()
  …

class Employee(Person):
  department = CharField()
  …

Two tables will be created and what looks like a simple query to the Employee child class will actually involve a join automatically being created. The same example with abstract = True in the Meta class allows abstract inheritance:

class Person(Model):
  name = CharField()
  …

class Meta:
  abstract = True

class Employee(Person):
  department = CharField()
  …

By putting abstract = True, the extra table for the base model is not created and the fields within the base model are automatically created for each child model. This avoids unnecessary joins being created to access those fields. This way of using model inheritance also avoids repetition of code within the child classes.

How to Install Apache Tomcat on Mac OS Sierra

Our previous tutorial on installing Tomcat on El Capitan had a lot of interest, so here it is an updated (but broadly similar) tutorial for MacOS Sierra.

In this tutorial, we will be using the open-source package manager Homebrew. If you’re not already using homebrew, check out its popularity on GitHub. It’s highly recommended for use with developing on Macs as it makes keeping track of installed software 100 times cleaner than doing it manually (everything is stored in one place, packages are easy to remove, upgrade and find configs for).If you don’t already have Homebrew, install it with:

First, open your Mac terminal window. If you don’t already have Homebrew, install it with:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

 

Step 1: Install Tomcat

Now we can easily install and track the version of Tomcat we’re using and config files etc with the following command:

brew install tomcat

This will take care of the downloading, installation and configuration of Tomcat and manage its dependencies as well. Take note of the output, brew commands are typically really good at displaying concise but useful info, errors and other help.

Homebrew keeps all installed packages (called “kegs”) in a “Cellar” folder – they like their beer references. By default, this is located in the /usr/local/ directory, so there should now be a tomcat folder listed by the command:

ls /usr/local/Cellar

Although an easier shortcut is available as:

brew list

To get more info on Tomcat specifically:

brew list tomcat

This should output something similar to:

$ brew list tomcat
/usr/local/Cellar/tomcat/8.5.15/bin/catalina
/usr/local/Cellar/tomcat/8.5.15/libexec/bin/ (15 files)
/usr/local/Cellar/tomcat/8.5.15/libexec/conf/ (10 files)
/usr/local/Cellar/tomcat/8.5.15/libexec/lib/ (25 files)
/usr/local/Cellar/tomcat/8.5.15/libexec/temp/safeToDelete.tmp
/usr/local/Cellar/tomcat/8.5.15/libexec/webapps/ (573 files)
/usr/local/Cellar/tomcat/8.5.15/RELEASE-NOTES
/usr/local/Cellar/tomcat/8.5.15/RUNNING.txt

The listed binary file Catalina is the “servlet container” used to run the Apache Tomcat server. This can now be started using the command:

catalina run &

The ampersand at the end makes the process run in the background, so after pressing the return key you get your terminal back but Catalina is still running in the background, you can remove it if you want to keep a dedicated window on the process.

Catalina can be stopped using:

catalina stop

 

Step 2: Configure Tomcat

Apache Tomcat comes with an inbuilt GUI management suite, however, for security reasons, this is disabled by default (to avoid dangerous default usernames/passwords). To enable the GUI manager, first edit the file:

nano /usr/local/Cellar/tomcat/8.5.15/libexec/conf/tomcat-users.xml
  • Nano is a text editor that comes with MacOS, although any will do.
  • If the file is blank or not found, you probably have a slightly different version of Tomcat installed, hit ctrl + x to exit, then run the command brew list tomcat and replace the above version number with the one you have installed.

Scroll to the bottom of the file and you should see several user entries surrounded such as:

<!–
<role rolename=”tomcat”/>
<role rolename=”role1″/>
<user username=”tomcat” password=”<must-be-changed>” roles=”tomcat”/>
<user username=”both” password=”<must-be-changed>” roles=”tomcat,role1″/>
<user username=”role1″ password=”<must-be-changed>” roles=”role1″/>
–>
</tomcat-users>

The <!– …  –> means these users are commented out. Leave them as they are and add a new entry after but above the </tomcat-users> line with the “manager-gui” role:

<user username=”someUser” password=”somePassword” roles=”manager-gui”/>

So the file now looks like:

<!–
<role rolename=”tomcat”/>
<role rolename=”role1″/>
<user username=”tomcat” password=”<must-be-changed>” roles=”tomcat”/>
<user username=”both” password=”<must-be-changed>” roles=”tomcat,role1″/>
<user username=”role1″ password=”<must-be-changed>” roles=”role1″/>
–>
<user username=”someUser” password=”somePassword” roles=”manager-gui”/>
</tomcat-users>

Obviously, you should use a unique username & password for security!

Now, start Catalina again:

catalina run
  • If you get errors, you probably need to stop it first, use catalina stop
  • By default, Tomcat runs on port 8080. There’s a really useful command to see what services are running on this a port: lsof -i :8080 (you may need to prefix with sudo for admin protected ports like 80).

Now if you go to the following page you should see a management GUI:

http://localhost:8080/manager/html

Here you can deploy .war files or exploded directories using the Deploy console and existing servlets are listed. You can even try visiting the already deployed servlets by appending the listed path to localhost:8080 (the default tomcat port):

So, the following should give the docs & some examples:

http://localhost:8080/docs
http://localhost:8080/examples

Enter the .war file location you wish to deploy and the desired path into the Deploy section and your application should now be listed as running with its path. Additionally, this can be used with an IDE like Netbeans or IntelliJ to run & debug servers using their configuration windows.

Any questions, please ask in the comments, and please share this article if you found it helpful.

PBCopy Mac command

A couple of useful Mac terminal (or iTerm) commands are the PBCopy & PBPaste. Here is a demonstration of how they work.

There are two basic ways to copy using pbcopy:

$ pbcopy < source_file.txt
copies text in source_file.txt

$ pwd | pbcopy
copies the output from pwd (the current directory) to the Mac clipboard

The copied text can now be pasted (either using menu’s or the CMD + V keyboard shortcut) into other applications in the usual way.

PBPaste works using the same syntax but in reverse:

$ pbpaste > destination_file.txt
pastes the current Mac clipboard to destination_file.txt

$ cd `pbpaste`
change directory to what's is currently stored in Mac clipboard

npm start error – TypeError: Path must be a string. Received undefined

This is a common generic error received whilst attempting to run older typescript based projects still using the now deprecated TSD package manager, on a system using TSD’s replacement Typings.

tl;dr  I just want to get this thing running quickly:
Installing (the deprecated) TSD package manager is often fastest way to get around these backward compatability issues:

$ npm install tsd -g

[disclaimer]using deprecated software is bad, see below for real fixes![/disclaimer]

 

Ok, now let’s look at actually fixing the problem.Unfortunately, the error’s output can be rather ambiguous, generic and difficult to get much helpful info from:

$ npm start

> node start.js
ts-loader: Using typescript@1.8
webpack built [...hash] in 1ms

ERROR in ./src/index.tsx
Module build failed: TypeError: Path must be a string. Received undefined
at assertPath (path.js:7:11)
at Object.dirname (path.js:1324:5)
at ensureTypeScriptInstance (/[...project src dir]/index.js:156:103)
at Object.loader (/[...project src dir]/node_modules/ts-loader/index.js:375:14)
@ multi main

 

Suspecting the error may be due to the project having been created using TSD rather than Typings, with the project having a tsd.json file rather than a typings.json file.

TSD has now been deprecated in favor of Typings, but the quickest way to get the project running is to install TSD, however this is obviously not ideal for anything but quickly checking out a project.

The Typings website provides info on how to upgrade from TSD to Typings, here:
https://github.com/typings/typings/blob/master/docs/tsd.md#upgrade

The official deprecation notice on the TSD GitHub page provides some more useful info and background too:
https://github.com/DefinitelyTyped/tsd/issues/269

A potential confusion is that the tsconfig.json remains the same name (although some parameters may need altering, moved file references updated/deleted), now the tsd.json is called typings.json with a similar config, but potentially also in need of manual editing after running the basic Typing’s upgrade process.

But hopefully, running npm start should work:

$ npm start
Compiled successfully!

The app is running at:

 http://localhost:3000/

 

If this does not work, a couple of additional steps may be required, including (but not limited to):

  • If a webpack compile errors say “Module build failed: Error: ENOENT: no such file or directory,” you should check your tsconfig.json file is not referencing any old tsd files which no longer exist (usually in the project’s typings/ directory).
  • You may need to create a tsconfig.json file in your project root directory (i.e. usually same one as package.json etc) if you do not already have one, visit the Typescript language homepage for more info. (Courtesy of https://www.garysieling.com/blog/fixing-webpack-error-path-must-string-received-undefined)
  • You may need to upgrade one or more old npm dependencies in your package.json file. Tools like npm-check-upgrades can help (use npm-check-upgrades -u to actually install the newer package versions – warning: may break the application).
  • Try suggestions from the above links, (e.g. the TSD deprecation announcement).
  • If all else fails, undo everything so far (if a git repo type:
    $ git checkout — .
    ) and install the TSD package manager (deprecated and bad, but a potential short-term workaround):
    $ npm install tsd -g

 

These package upgrade errors do seem to happen with a regular frequency, but are usually solvable with a bit of work. If the above solutions did not work for you or you have succeeded using another, please let others know about it in the coments.

Business Startup Advice

As we’ve found, being involved in startups, it can be a real help to have some expert advice to guide around problems. Recently an extremely talented friend of ours business strategist by the name of Jeannette Spaulding has helped us with our business strategy and so would like to recommend her advice to anyone else who wants help getting their startup idea running smoothly:

You can find Jeannette at http://www.positively.business

She has some incredible knowledge on the technical aspects of strategy & marketing which really helped some of our startups focus both making a positive impact on the world whilst also staying profitable in an ethical way.