Git Workflow for Sprint projects

It was a surprise to find a lack of clear diagrams for using Git properly in Sprint / Agile / Scrum programming environments. Therefore we thought we would take a moment to plot how we’ve always understood the best practice Sprint git workflow:

Git Sprint Workflow diag

As usual develop is the ongoing central branch, with master being the live, most stable branch which is only merged to via release branches and (when necessary) hot-fix branches for quickly fixing critical bugs on the live system. The key difference being that a new branch is created from develop for each Sprint cycle. Features / stories / tasks (here on known as stories) dished out to team members are then created as branches from that Sprint# branch.

If any stories are not completed within the Sprint timebox (e.g. 2 weeks) their branches can be carried over to the next Sprint as necessary. It should also be noted story feature branches can be merged, for example when a story relies on code from another story in the same Sprint. To do this run from the destination branch:

> git pull origin another-feature-branch

During the Sprint, completed stories / branches are tested and if all’s good a pull-request is created, but only during the Sprint Review at the end of the Sprint are the pull-requests (branches) merged back into the Sprint branch and then at the end of the review, the Sprint branch itself is merged back into Develop.

If a release is to happen after the Sprint (and it frequently should if following agile methodology) then the develop branch is tagged and a new “release branch” created from this tag. It is this release branch which is then merged back to master in what is in effect the release of a new version of your software (along with merging back to develop to keep everything in sync). The master branch can then be tagged to create an official “released” version, this tagging should also happen after hot-fix merges.

As is the general idea with Sprint workflows, a new release can be made after every Sprint (maybe skipping the occasional one if the software isn’t ready).  We usually make these new increments of the .# numbers, so x.1, x.2 … x.25, x.26 etc. Hot-fixes from master lead to  x.x.# increments, so x.x.1, x.x.2 … etc.

Often during release candidate testing minor bugs and tweaks will arise. Major bugs might require the abandoning of the release branch and new sprint work carried out but for quick fixes, a hot-fix branch can be created off the release branch. This hot-fix branch can be merged straight back into the release branch (and shouldn’t effect the release version number once released). 

Modern source hosting environments like BitBucket and GitHub make this whole process nice and GUI based, especially with inbuilt issue-tracking systems, however this can all be done purely from the command line, perhaps tying in with a standalone issue-tracker like TracZendesk or if you want to be really old-school, a simple Excel spreadsheet ;-).

Any questions or comments, please let us know.

Advertisements

Switching databases in Symfony upon application events

We’re big fans of the PHP Symfony2 framework and Doctrine2 ORM but a recent project involved a REST API for a multi-client system where each client wanted their database fully separated from each other, but we only knew which client database to connect to once a user had logged in (so no easy config.yml and parameters.yml configuration).

The login gets handled via Java Web Tokens using the excellent Lexik JWT Authentication bundle. So the first call to /auth/login_check with a username and password retrieves a token along with a id of the client (added using the onAuthenticationSuccessResponse listener documented here. Subsequent requests supply a HTTP Authorization header with the token and a custom “client” header specifying the id of the client (also documented in the above link).

This gave us stateless authentication for our API, but we still had to actually switch the Doctrine dbal connection to point to the right database. The solution (inspired by this Stack Overflow question) involved an onKernelRequest() event listener to switch the database from a base database (holds login / authentication data and is referenced in the default connection settings – or whichever connection you’re overriding – of your config.yml file) to the desired database and update the tokenStorage object accordingly. As a tip though, it’s best to keep the base and individual databases schematically identical as possible for reasons explained later.

First create the Event Listener:

Then add this to the services.yml file:

Then add the client databases to the end of the parameters-dist.yml file:

The security.yml file looks something like:

Conclusion:
This solution works fairly well in providing multi-database connectivity based on a site event (in this case user login) with the occasional bit of peering underneath’s Symfony’s hood. As mentioned above, keeping the base database schematically identical (i.e. same table structure, differing data) to your individual databases is very highly advisable. Some Symfony logic occurs before the onKernalRequest event gets fired (looking at your forms) so functionality limitations can otherwise occur there. Also parsers like the Nelmio Api Documentation bundle typically build pre-authentication (so whilst the base database is still selected).

Doctrine Migrations can also be enabled with a little tinkering (I may write a new post on how to do this if there is demand) but again, it helps if the schemas are identical (although isn’t vital).

Saving a root owned file in vim when not opened using sudo

A very common Linux administration annoyance can be opening a file in vim, making loads of changes, going to save and quit the file (using :wq) only to receive the message:
E45: 'readonly' option is set (add ! to override)
But as only root can write to the file, this won’t work either. Before pulling hair out, try this handy little command:

:wq !sudo tee %

Deleting folders beginning with a dash “-” on Linux

If you have a folder or file which starts with a dash (e.g. “-someFolder”) then simply running rmdir (or rm for a file) will produce the following error:

> rmdir -someFolder
rmdir: illegal option -- s
usage: rmdir [-p] directory ...

A neat way to delete such a folder is by referencing the directory first:

> rmdir ./-someFolder

The folder should now be deleted. Files are the exact same, just replace rmdir with rm.

Doctrine createQueryBuilder() – EntityManager vs. EntityRepository

EntityRepository and EntityManager have slightly different versions of the createQueryBuilder() function. Whereas the EntityManager’s version takes no arguments, EntityRepository’s version expects an ‘$alias‘ parameter. What is going on?!

The EntityRepository class wraps the EntityManager’s call, as such:

From /lib/Doctrine/ORM/EntityRepository.php

The function’s phpdoc reveals the reasoning behind the parameter count differences, EntityRepository returns a version of createQueryBuilder() customised for itself. We no longer need to specify the primary table we’re selecting from. Instead we must supply an $alias parameter which would usually be later supplied to the from() function.

Also note the above means all columns are selected from the entity by default.

Apt-get not finding packages

On a freshly installed Ubuntu system, apt-get may not have bothered to download its list of packages. This can lead to the confusing position of ‘E: Unable to locate package X‘ on almost everything. Fix this by entering:

sudo apt-get clean
sudo apt-get update

Packages should then be visible and installable via the usual command:

sudo apt-get install X

If you’re not sure on the package name, search for it using:

apt-cache search X

If you still cannot find the software you’re after you may of course need to add additional repositories.

Fun with counting in PHP

PHP’s loose typing can throw up some amusing bugs, take using scalars with the count() function. The function isn’t designed for scalars sure, but it doesn’t complain about them and often PHP makes use of global functions like count() rather than the more object orientated $i.count style, creating a nasty little minefield (one of many) for those unfamiliar with PHP’s peculiarities regarding proper objects vs. native types.

So the following looks valid enough:

$i = 5;
if (count($i) > 0) print "true";

But the code is bugged and whilst true will be printed as expected (maybe satisfying the programmer’s initial testing), guess what happens here:

$i = 0;
if (count($i) > 0) print "true";

We get an output of true !?#!??

What happens is the count() function evaluates one scalar value (the 0 integer) and so returns an integer of 1, which is greater than 0.

The following will print true:

$i = false;
if (count($i) > 0) print "true";

Whilst the following will not print anything:

$i = null;
if (count($i) > 0) print "true";

So remember, unless you want weirdness, only use the count() with arrays and objects implementing a Countable interface.

Quickly get memcached working in Python Django

As with most frameworks, the Django framework for Python can make use of caching to greatly improve performance for many common requests. Here we will look at using memcached as it enjoys good Django support and production use although there is also Redis support which definitely improves on memcached in some aspects such as data persistence.

  1. The first step is to install memcached on your server:
  2. RedHat Linux:

    yum install memcached

    Ubuntu / Debian Linux:

    apt-get install memcached
  3. Let Django know how to access memcached:
  4. In Django’s settings.py file, add the following line:

    'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache'
  5. Load the cache within your application
  6. from django.core.cache import cache
  7. Save the value to the cache
  8. cache.set('exampleValue',exampleValue)
  9. Retrieve the value from the cache
  10. exampleValue = cache.get('exampleValue')

The beauty being that exampleValue can be anything from a computed / database retrieved value to large blocks of static text or a URL etc.

The only problem with caches is they don’t always contain the data you expect, what if the value got flushed or hasn’t yet been stored? Lets rewrite step 5 to handle the event of the value not being available in the cache:

exampleValue = cache.get('exampleValue')
if not exampleValue:
     exampleValue = exampleValueLookup
     cache.set('exampleValue',exampleValue)

Here we see the value exampleValue being retrieved with a backup regeneration if the value has not been set. In a real application this would usually be encapsulated in a getExampleValue function or somewhere appropriate.

Symfony2 custom exceptions

When using the Symfony2 Framework it is often better to create custom exceptions to handle unique circumstances rather than incorrectly utilising one of the many pre-existing Symfony exception types. Using an interface improves code cohesion, allows the adding of custom functionality as needed and is super easy to setup.

The first step is to create a custom exception interface.

namespace AppBundle\Exception;

interface AppBundleExceptionInterface
{
}

Then create a custom exception class (perhaps in an Exception directory if it suits your project’s structure):

namespace AppBundle\Exception;

class NewTypeOfException extends \Exception implements AppBundleExceptionInterface
{
}

This exception can now be thrown and caught as desired, just be sure to name your exceptions well. You may be throwing them long after they were originally created!

The many different syntaxes of PHP’s if statement

The PHP language is blessed with many possible syntax structures for all sorts of language constructs. Some better than others but all have their place . Here we will be examining flexible usage of some of the styles available for use with if statements.

First, let’s look at the styles for the following statement in the standard c style:

$value = 1;
if ($value == 1) {
    print "if expression run";
} elseif ($value == 2) {
    print "elseif (1st) expression run";
} elseif ($value == 3) {
    print "elseif (2nd) expression run";
} else {
    print "else expression run";
}

(note: elseif has identical else if snytax in above and below examples)

In semi-colon syntax:

$value = 1;
if ($value == 1):
    print "if expression run";
elseif ($value == 2):
    print "elseif (1st) expression run";
elseif ($value == 3):
    print "elseif (2nd) expression run";
else:
    print "else expression run";
endif;

In switch statement format:

$value = 1;
switch (true) {
    case ($value == 1):
        print "if expression run";
        break;
    case ($value == 2):
        print "elseif (1st) expression run";
        break;
    case ($value == 3):
        print "elseif (2nd) expression run";
        break;
    default:
        print "else expression run";
};

The following are recursive language constructs.
In inline recursive format:

$value = 1;
($value == 1?
    (
        print "if expression run"
     ) :
    ($value == 2?
        (
            print "elseif (1st) expression run"
        ) :
        ($value == 3?
            (
                print "elseif (2nd) expression run"
            ):
            (
                print "else expression run"
            )
        )
    )
);

Using logical operators:

$value = 1;
$value == 1 and (print "if expression run")
or $value == 2 and (print "elseif expression run")
or $value == 3 and (print "elseif expression run")
or (print "else expression run");

 

Any other syntaxes missed?