iShamba: revolutionising farming in Kenya

iShamba is a farming information service, taking advantage of the popularity of Mediae's TV show 'Shamba Shape Up' to provide further support to farmers via their mobile phones.

The main users of iShamba are the call centre staff in an office in Nairobi, and the subscribing farmers who get regular advice, market prices and weather reports via SMS. The farmers can send SMS questions or phone the call centre to get more specific information. As we've already said, it's rare for us to produce something that doesn't have a big shiny homepage and lots of design to shout about. Nonetheless, the service is gaining subscribers and has recently featured in the Guardian. We're really pleased to be a part of it.

Some of the tech puzzle was clear from the outset—using the Django web framework we needed to: get information from vastly different external providers (weather forecasts, incoming SMS and phone calls, mobile payment, market prices); write algorithms for constructing weekly SMS messages to subscribers, with weather forecasts for their area, farming tips about their specific produce, and the latest prices of those commodities at their local markets; build the web management interface for Mediae; and build a real-time call centre web app. All within three months.

Regardless of how much planning you do, some issues only come to light once you start getting your hands dirty. One of the most satisfying technical changes we had to make was in handling the market price data we were receiving. Up-to-date prices are important to the service because there can be a significant difference in wholesale prices between markets just 60 miles apart. The Kenyan government organisation, NAFIS, was already gathering this data via people physically visiting the markets and manually recording the price for, say, cowpeas.

The problem with this approach is with what happens when that person isn't sure whether it's Cowpeas or Cow peas, nor whether to record the singular Tomato as a mass noun, or the plural Tomatoes for the countable individual fruits. And what about typos? We were receiving prices for Dolichos(Njahi (sic), White Irirsh Potatoes and Cabages. Nor were the names of the market towns immune from this treatment, and we needed to import hundreds of prices per day. This wasn't an occasion for human intervention, and we couldn't afford to discard data that might make an important income difference to the farmer receiving the SMS.

Enter fuzzywuzzy, a fuzzy string-matching library written in Python, which is essentially a user-friendly wrapper around Python's own difflib. Our typos were perfect material for fuzzywuzzy's extract function: we had a small number of pre-defined acceptable text strings, and they were sufficiently different that we could reduce the score cutoff without getting false positives.

Once we'd found a score_cutoff that worked, the convenience function extractOne turned out to be just what we needed:

Of course this doesn't just sit on its own. We're importing this data from csv files uploaded daily. To accomplish this we're already using the handy django-import-export application. It does most of the heavy lifting that you'd normally need to write when needing to upload csv data, with nice admin integration, but has customisable widgets for when your data doesn't match the vanilla field types.

Let's look at some code:

Note that both the commodity and market fields are using FuzzyLookupWidget. Let's take a look at FuzzyLookupWidget:

Note: that we're not actually specifying a score cutoff directly, because we want to be able to report a little more context if we do get a new commodity or market that isn't a good match. This keeps our error emails informative.

What does the end result look like?

No news is good news in this case.

Building iShamba has been a rapid ride so far, with much of the enjoyment of working on it coming from the variety of the problems we've had to solve. With a good proportion of our business logic dealing with external services, we knew to expect issues like this one, but we couldn't have known exactly what they would be. Of course, this is always the case in software development. The encouraging consequence is that as these problems are solved, the tools and workarounds become part of your arsenal for future projects. I'll be ready for the cabages next time.

If you'd like to find out more about iShamba, or want to discuss your exciting tech project, please get in touch. We'd love to hear from you.