Big data in government: the challenges and opportunities

Civil Service

February 21
09:42 2017

Ladies and Gentlemen, good morning.

Id like to begin by thanking Reform for giving me the opportunity to address you today. I want to talk to you about the potential of big data in government - and the hurdles we face along the way.

We often hear that this is the age of data, or that data is the raw material of a new industrial revolution. Theres truth in this. And theres huge opportunity.

Data can truly be a catalyst for a society, an economy, a country that works for everyone.

Of course, data isnt new. There has always been data. The Domesday Book is data. The Rosetta Stone is data.

But the rapid advances in technology and the development of analytical tools and techniques mean we can now gather and share data in huge quantities. We can process and analyse it at previously unimaginable speed. We can draw conclusions and create policies and services that reflect how people live now.

And we can help them live better, more securely, more healthily and more prosperously as a result.

Data entrepreneurs are mining public sector data to create apps and services to make our lives more convenient. Services driven by open data are already giving people more choice in where they get their healthcare, where they live and where their children go to school. Theres even a Great British Public Toilet app - a sort of relief map of the country!

In government, we get this. Weve always held enormous quantities of data - now we need to make sure we use it properly. Getting this right is the next phase of public service modernisation.

Thats why this month we have published the Government Transformation strategy. And the Digital Economy Bill is in the last stage of its journey into law.

There are 3 key areas of opportunity that we need to grasp:

  • first, improving the experience of the citizen
  • second, making government more efficient
  • third, boosting business and the wider economy.

The impact of data analytics and big data in our lives for example the way online retailers tailor their recommendations for the food, books and music we buy - is quite familiar.

Less has been said about the transformative power of this technology for the delivery of high-quality public services. And its time that changed.

With the evidence of data we can spend less time developing policy and services that dont work, and instead focus on continuously improving those that do.

I want people to turn to digital public services as readily and confidently as they do when shopping, socialising or checking bus times.

By doing so, we can actually change the way citizens interact with us - making the relationship we have with them more transparent, more responsive, and based on increasing levels of trust.

For big service delivery departments like the Home Office, HMRC and DWP, data analytics means the ability to search across organisational data sets. It can provide data for operational teams to put into practical effect.

In DWP, for example, providing job seekers with more targeted advice, and opportunities that closely match their personal profiles. The department is also working on data-informed tools, such as interactive visualisations of benefit claimant trends.

There are examples at home and abroad where data is being used to address peoples real concerns about their daily lives; providing solutions that were not available before.

In June last year, for example, Land Registry and partners published the first UK House Price Index, and provided a single source of information as opposed to the multiple competing versions which existed before.

Land Registry data has also been used to create a range of information services. From whether rude-sounding street names have an impact on house values - (they do!); to more serious matters, such as whether your home is on a floodplain.

Land Registrys Flood Risk Indicator service uses data from the Environment Agency to identify flood risk for any registered piece of land within England and Wales.

The Companies House Service gives us free access to real-time information on companies. Its receiving millions of search requests every day from people checking supplier and customer information.

The service can also be used for more mundane but practical reasons - if youre getting in builders to do work on your home, you can go on the Companies House website and check them out first.

Healthcare is another exciting area. Moorfields Eye Hospital and DeepMind Health are partners in a research project that could lead to earlier detection of eye diseases.

At the moment, clinicians rely on complex digital eye scans. 3,000 of these scans are made every week at Moorfields. But traditional tools cant explore them fully, and analysis takes time.

Moorfields will share a data set of one million anonymised scans with DeepMind, who will analyse them using machine-learning technology. This can detect and learn patterns from data in seconds, to quickly diagnose whether a condition is urgent.

With sight loss predicted to double by 2050, the use of cutting-edge technology is absolutely vital. The right treatment at the right time can prevent many cases of blindness or partial sightedness. Up to 98% of sight loss resulting from diabetes, for example, can be prevented by early detection and treatment.

Analysing data can also play a direct and powerful role in protecting the most vulnerable in society.

The Home Office Child Abuse Image Database has transformed the investigation of child abuse crimes and child protection. It won the Civil Service Innovation Challenge in 2015.

The database brings together all the images of abuse that police find. Using the images unique identifiers and metadata, they can check devices theyve seized from suspects against the material on the database much more quickly.

Previously a case involving, say, 10,000 images, would typically take up to three days to review. Now, it can be reviewed in an hour.

So, we have a process that is cheaper, less labour-intensive and more efficient. This is all good. And it makes the investigation and prosecution of these appalling crimes vastly more effective.

There are also examples of government data meeting needs that more of us will be familiar with - like tax.

Personal tax accounts from HMRC now take a real-time digital approach. For the first time you can log in when you like, check your tax information and manage your details online in one place. More than 8 million citizens have now signed up, including some of you here today, I expect. HMRCs digital team now has around 30 new online services in development.

Government open data, combined with digital technology, can also fuel an open economy. It will provide information that entrepreneurs, data start-ups and the general public can use.

In 2015, the digital sector contributed 118 billion to the economy, supporting over 1.4 million jobs.

The UK Government was an early world leader in open data. So far, weve released over 30,000 non-personal data sets in machine-readable formats, for no cost, and open for anyone to use or build upon. This has enabled the creation of innovative products that deliver value for citizens.

So far, these data releases have been turned into over 400 different apps. You may well have used some of them yourselves:

  • the Floodalerts API: which uses Environment Agency data to provide 15-minute updates about flood risk
  • UK Food Hygiene: which lets you see take-away and restaurant food hygiene ratings to help you make decisions on where to eat out
  • FillThatHole: a site for reporting potholes and other road hazards across the UK using ONS Census geography data
  • There are also apps for finding the best dentists, GPs, schools and universities.

The list of sectors tapping into Defra group data from Lidar (the airborne, laser equivalent of radar) is truly remarkable. British wine producers are using the terrain-mapping data to help them decide where best to plant vines, and if the recent prominence of English sparkling wine is anything to go by - they are having great success!

Architects are using it to build a model of London as they plan the next high-rise building; computer game developers to build new landscapes for Minecraft; and archaeologists to discover lost networks of Roman roads. In October last year alone there were almost 21,000 downloads of Lidar data from

Some companies dont only use open public data to build a business but also to act as a positive disruptive force. FoodTrade, for example, maps the food supply chain system, making it easier for people to buy and sell fresh local produce.

And other start-ups are using open data in ways that boost the economy by providing data analysis tools and data products that support the growth of SMEs.

A firm called GeoLytix offers a range of products based on geospatial data - giving smaller companies access to information that they would not be able to do on their own, and helping them to solve business location issues in the process.

So as we look to improve the availability, quality and use of government data as the basis for fully transformed public services, it will also provide a new stimulus for data-based businesses.

Because government data is public data we have a duty to use it well and open it up where possible - and we have to be seen to do so cost-effectively, efficiently, proportionately and appr

Related Articles


  1. We don't have any comments for this article yet. Why not join in and start a discussion.

Write a Comment

Your name:
Your email:

Post my comment

Recent Comments

Follow Us on Twitter

Share This

Enjoyed this? Why not share it with others if you've found it useful by using one of the tools below: