Django: a guide to upgrading your application

The tech team at Torchbox have used Django since its early pre-1.0 days. Many of the applications we've developed have been used for many years with the ongoing addition of features. So, it's important to keep them up to date with the latest that Django has to offer - new functionality, improvements to existing ones and, of course, continuing security updates.

Django has settled into an eight month release cycle. Having an efficient upgrade process helps make this a smoother ride, encouraging it to happen on each point-release. In this blog post I look at the process we use for our bespoke applications, which has been refined by the experience of each previous upgrade.

The aim of the process is to have the application running functionally the same after the upgrade as it was before. Once that's been achieved the benefits of the new version can be exploited. 

Unlike open source Django packages (such as our own Wagtail), with our bespoke applications we have the luxury of not needing to make the codebase simultaneously compatible with both the latest and previous versions of Django; we can burn our bridges behind us. Multi-version support presents its own challenges for open source packages, nevertheless, many of the tips in this guide still apply. 

The starting point is a good read of the Release Notes (e.g. 1.8). As well as showing all the new toys the latest release has offer it details the particular changes relevant to upgrading the application: backwards incompatible change; features deprecated; and features removed.

The Release Notes are usually pretty long but it are well worth reading through them carefully and making a checklist of any that you know will apply to your application. This checklist will help when it comes to making changes to your codebase. You probably won't notice everything - in every release there's usually some seemingly minor change that you don't think will be relevant but ends up being the reason for some strange exception or regression. But, the penny usually drops quicker if you half-remember a change you read.

Unless your application is small, or you have a penchant for reinventing the wheel, it probably has dependencies via third-party Python packages - typically Pip-installed from PyPI. This is a good point at which to take a look at the package version you're currently using and see whether it's already compatible with the new version of Django, has known issues, or has a new version that is compatible. 

Whilst you can upgrade your own application's code that won't help in having a production ready application if any of your dependencies are show stoppers. But what if a package's maintainers are slow to update it? Other than sitting it out, you could fork the package and update it yourself (ideally then submitting the changes back via a pull request). If the official package doesn't get updated it might be time to bite the bullet and look for an alternative, better-maintained package (you're then also less likely to face the same trouble on the next Django release).

The next waypoint is getting runserver to start the application without any errors. Most error/exception messages you'll see are descriptive enough to match to backwards incompatible changes or feature removals documented in the release notes. Tackle one type of issue at a time, fixing it at the point in the codebase where runserver failed, but also wherever else it occurs (bring out your favourite grepping tool). 

Even if you miss a few instances initially, it's helpful to put all the fixes of one type of issue into one commit to your version control system. Including some of the Release Notes text on the issue in the commit message helps to document your upgrade, e.g. "Django 1.8: select_related() now checks given fields". Also, tick them off the checklist you made. Rinse-and-repeat until runserver starts up your application. Hopefully, by this point, you'll have addressed many of the prominent backwards incompatible changes or feature removals.

In all likelihood though, soon after you start using the application you'll come across other errors. Again, unless your application is small you probably don't want to manually check all of it. This is where you pat yourself on the back for having maintained a test suite with good code coverage of your application. So, as with runserver, run your test suite through and group the failures/errors by type of issue. Then tackle each issue that came up. In some cases that'll involve updating the tests themselves - partly for their own code and partly for what they're testing if some functional change has been necessary in the application. Eventually, your test suite should run through cleanly.

(If you're upgrading using a beta/candidate release of Django and a new beta/candidate/final comes out it's important to test again in the latest version and not just update to the latest point-version. It may well come with changes that break your code, sometimes in subtle ways.)

Look back over your checklist to see if there's anything on it that you haven't already addressed (which might indicate a gap in your test coverage that could do with filling).

If your test coverage is good then your application should be more or less compatible with the latest version. If you have the time and resources, this is a good point to steal a march on your future self and resolve Django deprecations in your codebase. Deprecations are things that will change or be removed in future versions of Django - typically two releases ahead. So, with Django 1.8 you'd be looking for things deprecated in Django 1.7 that will be removed in Django 1.9, and things deprecated in Django 1.8 that will be removed in Django 1.10. Tackling deprecations early rather than waiting until the release that forces it on you has the advantages of avoiding more deprecated code being added in the interim and, in some cases, earlier use of the benefits they bring. Your future self will thank you. Due to the roadmap changes to deprecations there is now significant long-term benefit in ensuring that your Django 1.8 upgrade is deprecation-free.

One way to identify deprecated code is to re-run the test suite while enabling Python's warning control - Django will emit deprecation warnings when your code hits them and Python will print them in your test output. By setting minimal verbosity in the test runner it's easier to see the warnings. So, for example, rather than running:

./manage.py test 

you'd instead run:

python -W all manage.py test --verbosity=0

By sorting through the warnings, identifying those that arise from your codebase (some will be from third party packages and Django's use of its own components) and grouping into the same type of deprecation, you'll have a list you can work though to fix.

If you pin the versions of your Pip packages, chances are they haven't been updated for a while on the if-it-ain't-broken-don't-fix-it principle. A Django upgrade can be a convenient time to bump packages up to their latest versions to gain the advantage of any performance, security or functionality improvements. Tools to help with this is are pip list --outdated or the pip-review package. These inspect your Pip packages, check PyPI and report your version and the latest version available. It's worth looking at the release notes of packages you intend to update, particularly if it's a big version bump, to see if there are any backwards incompatibilities that might affect your codebase. After updating any versions, run your test suite again to check for regressions.

As a belt-and-braces check manually test critical components of the application as a real user in case that reveals errors that unit or Selenium tests might have missed.

Once I'm happy that all seems well in my development environment I deploy the upgrade into a staging environment where it's soak tested for about a week. This sometimes reveals obscure issues that come to light in near-production usage, but more typically gives increased reassurance that no major problems are to be expected when, finally, the upgrade goes into production.

And once it's in production? Well, if you're really keen, forewarn yourself of headline changes in the next Django release six-months or so hence (e.g. 1.9). More enjoyably though, make use of the new Django features now available to you (that hstore-backed, Jinja2-rendered feature you always wanted in your application is now in your grasp)!

1. Read the Release Notes, make a checklist
   
2. Review third party package compatibility
   
3. Get runserver starting cleanly
   
4. Get the testsuite passing
   
5. Clear your checklist
   
6. Remove deprecations
   
7. Update third-party packages
   
8. Manually test critical components
   
9. Soak test in a staging environment
   
10. Deploy