Putting Public Open Data to Work…?

A couple of posts out today on the Guardian Datablog review the progress of the UK’s open government data project to date (Government data UK: what’s really been achieved?, Nigel Shadbolt: A year of data.gov.uk).

One of the pieces of the jigsaw that I think has been largely ignored in many of the discussions I have seen around open data is the extent to which open data is incorporated into the productive workflow of an organisation, rather than just venting the data exhaust of an organisation every so often and pretending something useful has been released…

One of the reasons commonly given for why organisations should open up their data is the idea that a more effective use of your data may well be found by someone else: for example, by identifying previously unimagined ways of unlocking value, or exploiting network effects that arise from being able to merge one dataset with another.

But it seems to me that the data should also be useful to the organisation that released (otherwise, why collect it?), and that one way of making the most of open data stores is to put the data to work, by requiring day-to-day users of the data to access it via the datastore (an “eat your own dog food” argument…).

If data is collected and reported on within an organisation, then consideration should be given as to whether the workflow associated with that data might pass through an open data store, or whether the open data store might provide a view over data as it passes through that workflow. Where data is reported in a public (and maybe even just FOIable) way, submission of the report – for example, to a Ministry – might pass through the open data store. That is, rather than local gov sending data based reports to central gov, central gov picks up those reports from the local gov’s open data site.

That is: in cases where public institutions are currently expected to push open data and reports based on open data to other public institutions (or maybe other internal departments), they should start pushing that data to public/open datastores, and let the other party pull the data from that public data store.

What this means is that the business of data is, wherever possible, is mediated through open and public datastores. [RELATED: list of local government data burdens]

Another approach that I think might hold promise is for local councils to develop on their own, or in partnership with other councils or private enterprise, data driven sites in areas where they have a particular specialism or interest (which may include a potentially commercial interest), and then offer these services under a subscription basis to other councils across the UK and in so doing develop a national reach. For national scale delivery, it might be that this is handled by a commercial partner, with the original developing council taking a commercial stake and getting a return.

The national aggregation of local services idea is worth bearing in mind because we shouldn’t necessarily expect a user of a council data powered website delivering location based or location related services to know which council a particular location falls in.

For example, (and I know this isn’t a council service…), something that is done locally but at national scale is blood donation.

The NHS Blood donor service site provides a means of identifying the time and place for local collection sessions. It may well be that the the data is generated on a regional basis, but why should there be any more than a single place to go to find out this information?

Here are some more examples of services started locally that might scale:

RateMyPlace – food inspection ratings, currently for a handful of councils in the Staffordshire area.

Who Owns My Neighbourhood? – identify council owned land and buildings, currently in the Kirklees area.

Your Ceremony – venue hire and registrar booking for civil ceremonies, via East Cheshire council.

It’s not just councils who might initiate these vertical, cross-council vertical data or service sites of course. For example, FixMyStreet is a MySociety site for reporting local issues relating mainly to the built environment, such as potholes. FixMyStreet engages citizens in the human level instrumentation of local neighbourhoods, changing attitudes as it does from “the making of complaints” to “the reporting of issues”. FixMyStreet also offers other issue tracking features such as the ability for issues to be “closed” when they are fixed. I don’t find it hard to imagine that councils might pay a small subscription for additional reporting and management tools built around the FixMyStreet workflow, although the practicalities of that might be very different! The FOI requesting site What Do They Know could also be seen in a similar light as a workflow for managing FOI requests.

Architecturally, there may be several different approaches to the design of these sites, and how to engage with them. For example, if a site has a write API, or can import data from a variety of document formats, albeit ones structured in a particular way, a council might easily write or upload data that can be normalised and presented in a consistent way by the aggregating site.

For councils that do publish data, but in an inconsistent way, aggregators such as OpenlyLocal attempt to scrape the data and normalise it to provide a uniformity of access to data, in a consistent from, across council regions.

Where aggregation sites normalise data and represent it via an API, it provides an opportunity for third party developers or vendors to invest in the production of single application that can be configured to present localised views over the data to individual councils, or allow council developers to share widget code with other councils.

Sites that aggregate local data at national scale are thus beneficial in several ways:
– they provide consistency of experience for folk who move between areas and mask boundaries between the sources of data (which is fine, if everyone os reporting in a consistent way…);
– they minimise the need for a user to know which council area they are in;
– they provide the ability to compare activity across neighbouring regions;
– they provide the opportunity for additional services operating at national scale to make use of the data.

PS if you know of any other councils that have developed vertical sites, ideally data driven ones, such as RateMyPlace or Who Owns My Neighbourhood, that could scale nationally and mask council borders, please add a link in the comments:-)

A good place to look for new services may be workflows where there is the production of standardised reports, e.g. to central government or using standard procedures, and perhaps more revealingly, standard forms! That is, if every local gov inspector across the UK users form bZ23/A to file a particular sort of report, it could suggest that a data driven service around that data might scale to a national level… or not;-)

PS for a corollary to this – data horizontals with a local focus, see Greg Hadfield on local news sites “re-inventing themselves as local data hubs” (Open-data cities: a lifeline for local newspapers).