Learning how to improve data capture and analysis from surveys

To understand how hauliers move goods around the UK and internationally, the Road Freight Statistics team at the Department for Transport (DfT) carry out surveys throughout the year.

These surveys are done on paper and are completely offline.

This creates a host of problems: from incomplete or incorrectly filled out forms, to illegible handwriting.

It’s mandatory to complete the survey, which leads to issues of frustration for those filling it out as they’re not sure as to its purpose and are not motivated to make sure they’re doing it right.

While response rates look high at 90%, in reality far fewer of the completed surveys contain usable data.

The challenge was to develop a digital service to simplify the process for survey respondents, improve the quality of the data captured and explore the potential for machine learning.

An alpha project sets out to prototype a solution and work quickly to validate whether it works or not.

Our riskiest assumption was that we could make a digital version of the survey and that the survey respondents would prefer this. It’s a helpful method for incrementally shaping an effective solution, without the risk of becoming wedded to a particular approach.

For this 8-week alpha, we created a multidisciplinary team of 10 from DfT, Torchbox and a partner agency Lagom, spread across 5 locations. This blended team approach allows for the drawing together of knowledge from many areas and people, which is fundamental to being able to solve complex problems.

By mapping the existing paper forms, we were able to identify pain points and opportunities for improvement.

Using GOV.UK patterns enabled us to create realistic prototypes to test with users during the first sprint. Their feedback quickly revealed subtle design challenges in the digital workflow.

For example, on the paper forms it is easy for a user to leave a section blank if the questions don’t apply to them, and this is something the digital workflow also needs to allow. Unfortunately, it is just as easy for users to leave a section blank by mistake, and we needed to prevent this from happening.

We also identified opportunities to streamline the survey by dynamically adapting questions based on data already captured. For example, we can omit a question about trailers if we already know that the vehicle being driven can’t physically carry one.

One of the advantages of moving to a digital service is the opportunity to exercise more control over the way data is entered to improve data quality. We worked on ideas for ‘soft’ validation that would give users helpful prompts: for example, by suggesting that a response might be incorrect. In this way we can encourage users to ‘sense check’ their answers without preventing them from going to the next step.

This alpha identified many ways that a digital service would improve the experience for hauliers. We can make it quicker and easier to complete, while also making sure that the answers they give are more accurate and complete, which improves the data collection.

The teams in DfT who support those users, process the data and carry out statistical analysis also benefit. Our alpha identified ways to use machine learning to automate even more of the process.

This is particularly promising in relation to the categorisation of goods, which is currently coded to an EU mandated standard, called Eurostat. We tested using sample data sets and were able to show that by implementing this technology, it would provide enough accuracy to reduce the cost of staff time spent on manual categorisation.

• Improved data capture: addressing issues such as incomplete or incorrectly filled out forms
• Quicker and easier survey completion: reducing frustration for hauliers
• Machine learning: using automated categorisation for quicker and more accurate analysis

This alpha passed its assessment, meaning our research was sound, the insights validated and the prototyping robust enough to take to Beta.