OUseful.Info, the blog…

***”1. How do you think Government should best balance its objectives around increasing access to data and providing more freely available data for re-use year on year within the constraints of affordability?”

[Paras 1.12, 1.17, 1.18] Presumably, the first implication is that the PDC will incoprate public bodies involved with the production of “core reference data”/those organisations “whose primary purpose is collecting, managing and disseminating data and providing value-added services based on that data” and the policy framework for deciding who’s in and who’s out must in the first instance rule HM Land Registry, Met Office and Ordnance Survey in? Based on the criteria used to rule these organisations in, is the gut feeling that organisations such as Companies House (who mint unique corporate identifiers), the General Register Office, Office for National Statistics, DVLA (eg Vehicle Checking or Driver Validation Service), Highwyas Agency (eg http://www.trafficengland.com/index.aspx?ct=true ), academic data repositories such as http://www.data-archive.ac.uk/ or http://www.census.ac.uk/ , The National Archives, the data models and data assets being developed as part of the BBC Digital Public Space project or the JISC UK Discovery initiative. With publicly funded research increasingly being required to disseminate findings through open access publications (eg http://www.epsrc.ac.uk/about/standards/researchdata/Pages/default.aspx ), to what extent might (or should?) the deposit and/or release of research data be covered: a) by open data principles, b) via the PDC or a research council equivalent, bearing in mind that access to publicly funded research data may be subject to FOI requests (eg http://www.jisc.ac.uk/publications/programmerelated/2010/foiresearchdata.aspx ).

["1.22. The way that Government has sought to cover those high fixed costs and to ensure sustainable investment in data infrastructure has been to encourage public sector bodies to licence their core reference data to third parties"]

But costs introduce friction downstream and may result in one part of government (the data publisher) recruiting more than cost from other public bodies? In addition, is it possible that central data collection bodies may as part of their remit collect data from other public bodies that is then resold back to those bodies in an alternative (albeit potentially enriched) form?

PDC Objective 3 states: “create a vehicle that can attract private investment.” So there is presumably a requirement that money flows in from the private sector to the PDC and then out again in spades (because investors will want a return)?

[4.12] If there is a large up-front cost in producing/releasing data, and limited marginal cost, another model would be a one-off fee to offset the production/release cost, rather than a metered/ongoing usage fee offset against production/release cost + marginal cost? That the public body would carry the marginal cost is just a consequence of its data being worked/used, which is partly the point of releasing it in the first place? (ie presumably some benefit accrues elsewhere in the system as a result of the data being worked?)

[4.22] A single fee may provide a barrier to entry to personal users, researchers, SMEs engaged in invention and innovation where there is no established market for as yet undeveloped products or services. Might fee waivers be a possibility, and if so, how would they be awarded. Might there be an equivalent of a public lending library service (a service that traditionally has provided universal access to information, including information from resources that may have a singnifcant acquisition cost associated with them) that will provide “personal research” access to a public task dataset?

[4.23] Is the work of producing datasets part of the public task of the Office for National Statistics, and if so, will we have to pay for access to those statistics?

[4.24] As a corollary to the case, for example, of locals councils licence out the management and operartion of civic carparks, would the public bodies be allowed to do the same with contracting out the management, publication of and charging for their public data usage by third parties, and if so, how will limits be set on the pricing, bearing in mind any commercial operator would expect to make a financial return on the operation of that service, and would it imapct on the way the public body collects, quality checks and operates its own data processes?

If public bodies are to develop “commercial products to serve commercial markets”, how does this sit with para. 4.18 (profit maximisation model) where an “incentivised PDC [would] fully commercialise all its products and services. While aligned with a strategy focussed purely on maximising value for the taxpayer such a model is unlikely to be consistent with Managing Public Money guidance and delivering on a commitment for free data”? Presumably the “commercial products to serve commercial markets” would be expected to be profit maximising, or not? Cost recovering (as in 4.23)? But what cost (eg would that include the cost of advertising, marketing, and other activities associated with commercial services)?

[4.25, 4.26, 4.28] Freemium does not necessarily imply “try out”. Many freemium services provide an access quota that allows an on-user to use the service as part of their own service, for free, up to certain usage limits. If the usage is heavy, then the commercial plan kicks in. But the small player can run a small service, for free, until they hit usage limits. In some cases, a condition of using the freemium service may be that the user cannot cache the data; ie they must faithfully draw the data down as they use it, rather than building up a local copy. In other cases, they may be encouraged to cache the data so as to prevent repeated service calls for the same data, in which case they usage quota is based on unique data accesses rather than repeated data accesses.

***”2. Are there particular datasets or information that you believe would create particular economic or social benefits if they were available free for use and re-use? Who would these benefit and how? Please provide evidence to support your answer where possible.

***”3. What do you think the impacts of the three options would be for you and/or other groups outlined above?”

[4.39 Government as user of PDC data] If the fees go up, and public bodies are changed universal commercial rates, they will have to pay more, which will introduce further friction into the process and reduce opportunites for effective data (re)use.

How I read the “options”:
["4.40. Under all options, charges for some units of PDC information are likely to change, with more data being provided free at the point of use."] So there are no additional benefits from Option 1.. so rule this one out?
["4.41. Under Option 2, it is possible that some efficiency savings could be delivered through having a single price, although there will be some upfront investment and resource required to implement a change."] Savings possible, but it will cost in the short term? Rule this one out too?
["4.42. Under Option 3, it is likely that in the short term income would decrease, but if the freemium model was successful income might then increase over time."] Presumably we’re expected to read this as: “It won’t cost anything, and profits may go down in the short term; but then we might get a viable business out of it, and moreover a business capabale of growth, using a sexy sounding techie inspired business model… Cool… let’s have that then’? The truth being, of course, that costs are generally associated with any change, and that this is a status quo offering, where public bodies charge other public bodies and private enterprises for data collected as a matter of course (although admittedly at some expense) as part of the operating environment for government.

***”4. A further variation of any of the options could be to encourage PDC and its constituent parts to make better use of the flexibility to develop commercial data products and services outside of their public task. What do you think the impacts of this might be?”

["4.30 There is the potential for providing a PDC and its constituent parts with greater encouragement to make better use of the existing flexibility to develop commercial products to serve commercial markets."] Does this include the ability to develop commercial services based around expertise and support? (Expertise that may not be available widely, for example, particularly to SMEs? In which case, the service would also help support knowledge transfer from the public sphere and into the private sphere?

Rather than produce data and make it available to other public bodies as well as developing commercial products, would it be possible to give the data away under a truly open license and task the PDC with developing data products and services that save the other public bodies money, working with them to reduce costs that can be then considered as in kind direct returns on investment in the data services and products.

***”5. Are there any alternative options that might balance Government’s objectives which are not covered here? Please provide details and evidence to support your response where possible”

[4.10] The assumption here appears to be that payment for data should be based on the basis that commercial users purchase a license to make use of data from the PDC and pay the PDC directly in financial terms. However, might a commercial user not offer an in-kind payment, such as a guarantee to resell services /at a discount or reduced margin/ to other public services, or make value-added versions of the data produced by the commercial user available for free to specified public bodies? This compares with 4.17 where data is provided free to users who then resell added-value data back to the public body. What is important is that if PDC bodies are producing value add data, this should be provided free of rights encumberance to other public bodies, and ideally free of cost; the issue then remains of how the value added data may be passed on to non-public bodies? The intent of these users might also be worth considering: for example, personal or academic research, commercial research/innovation by SMEs, or as part of a service offering by an established larger company. Differential license/charging agreements of course need to be fair, but might this not be handled through offset grants, for example public data access grants awarded to SMEs via the TSB?

[4.36] Defining the future PDC on the basis of supporting incumbent business models predicated on current processes and ‘the old way of doing things’ is a dangerous step to take. If the open data policy framework is intended to foster innovation, it would be foolish to constrain innovation and limit the future possible use of open and public data to legacy models and processes that represent the current status quo. True innovation may well be disruptive, and upset the current status quo. Such is life.

A set of models that do not appear to have been considered are business models that develop around open source software. In the same way that data can be expensive to produce, may be protected by ownership and licensing rights, and may be used as the basis of other commercially viable services, so too can software. A summary of business and sustainability models appropriate for the open source software domain can be found here: http://www.oss-watch.ac.uk/resources/businessandsustainability.xml It may be worth doing a simple mapping of these models, based as they are around open source software, onto an open data (rather than software) resource context. There is the tension that whilst cost recovery by selling on data may be deemed to be an acceptable mode of operation for a public body, in part because it supports cost recovery through getting a return on sale of goods/services for minimal marginal cost of making those goods/services available, the sale of high value consultancy, for example, requires large additional cost and activity not aligned directly with the provision of public service (in effect, the use of public service to provide private commercial services outside the public sphere, not just internally on a cost recovery basis).

If it is the case that better access to information – and data – helps us make better decisions (and I’m not convinced that what we want is to make decisions: most people have no real choice and just want effective local public services), then the reward to the public body is not so much a direct financial return as a minimisation of costs incurred elsewhere becuase a bad decision was made.

Recent years have seen a return to prize fund/Grand Challenge based funding models in wich prizes are awarded to technology solutions to particular technical challenges. This funding models replace the research funding support model with a reward based model. To what extent might the PDC act as a prize fund awarding body that can reward innovation around the use of public data, and sponsor parties engaged in such competition with “data permits” or “data credits” that provide them with data access in return for them submitting responses to data related Grand Challenges?

To what extent might the Technology Strategy Board ( http://www.innovateuk.org/ ), maybe under the auspices of the Small Business Research Initiative (SBRI) [ http://www.innovateuk.org/deliveringinnovation/smallbusinessresearchinitiative/whatissbri.ashx ] work with SMEs to provide “data credits” that provide companies with access to PDC content if it must be otherwise made available for a fee?

Could the TSB, in association with the PDC, even operate as an angel fund, supporting companies wishing to develop services or products based on public data, in exchange for a share in the companies involved, harking back to ideas behind the foundation of 3i, for example?

***”6. To what extent do you agree that there should be greater consistency, clarity and simplicity in the licensing regime adopted by a PDC?”

Experience of using Creative Commons licences and open source software licenses suggests that even within a open licensing framework, if different license types are combined it rapidly becomes difficult to work out how license conditions surrounding differently licensed components interact. Haviong a single license mitigates against creating confusion through complex, and possible inconsistent, combinations of license conditions arising from the novel combination of differently licensed resources.

The confusion as to what is allowable may act as a significant barrier to developing services that combine resources licensed in different ways. Since much innovation is likely to arise from combination of resources, the multi-license approach is not really viable. Regulating on how datasets may be used/what license terms apply for different use cases may place arbitrary conditions on the innovation of new models that fall outside or across models that are assumed to be possible when the model license conditions are framed.

Furthermore, in a truly open licensing regime, the scope of reuse would not be artificially bounded and the user would be free to reuse the resources in any way they wanted.

***”7. To what extent do you think each of the options set out would address those issues (or any others)? Please provide evidence to support your comments where possible.”

Options 1 or 2 may lead to situations where complex and even pathological combinations of different license types make it impossible for a user to work out whether or not they are allowed to combine a set of resources in a particular way, or develop business models that operate across different license condition regimes.

***”8. What do you think the advantages and disadvantages of each of the options would be? Please provide evidence to support your comments”

Option 1 “[5.10] … each organisation within a PDC would have its own portfolio of standard licences, terms and conditions appropriate to the nature of their business.”

Option 2: Overarching PDClicence agreement, subject to: “[5.17] flexibility to add additional schedules where necessary underneath that overarching agreement.”

Complications around from ill specified consequences arising from the combination of differently licensed resources arise here just as they do in the case of option 1.

Option 3 “While a single licence would offer greater consistency of standard terms and conditions it is likely that there would be a wide range of other terms, clauses and schedules required to cover the various types and uses of PDC information. It is therefore likely to be lengthy and will contain clauses and schedules that will not be relevant to all users”

Does this mean that there will essentially be different licenses according to the status of the user (personal use, academic, commercial, etc) rather than the situation in options 1 and 2 where there are essentially different licenses relating to the use to which resources will be put?

***”9. Will the benefits of changing the models from those in use across Government outweigh the impacts of taking out new or replacement licences?”

I don’t know.

***”10.To what extent is the current regulatory environment appropriate to deliver the vision for a PDC?”

“[6.8] … it is envisaged that all organisations within a PDC will be advised to develop and agree with the regulator the statement of their public task.”

So the management of the current operting funds will be expected to work together to produce their own regulatory framework, at least insofar as the definition of their public task goes? This is likely to be backward looking and protective of current operating models rather than being open to new models and potentially even new ways of defining public tasks that do not respect current organisational boundaries, processes and modes of operation. The PDC as thus described is a way of bringing together current orgnisations and their associated business models and allowing them to work together to protect those interests, interests that were defined to support a data environment that may no longer exist.

“6.10. In the freemium model there may be a role for the regulator, as indicated earlier, in advising PDC bodies how they can best go about making practical arrangements to make more data free for re-use while ensuring a sustainable business model.”

Requiring that any innovations also protect the current operating model suggests that the establishment of the PDC is actually a rearguard action to protect against a radical change in the ways in which data is produced, managaed and exploited within government, as well as for the wider public good through development of third sector and even private services.

***”11.Are there any additional oversight activities needed to deliver the vision for a PDC and if so what are they?”

The vision being the preservation of the status quo through the creation of a conglomerate of current data selling public bodies? And through oversight, you presumably do not mean the creation of a body that can force through changes to the way the board of the PDC decide it will operate, but rather will limit it’s role to seeing that the board does what it says it will do?

The current proposal for a PDC seems to favour the creation of a conglomerate charged with exploiting public data for financial return wherever it can, rather than act as a regulator, ombudsman, or advocate tasked with getting the most value out of public data through making effective use of it, and maximising the possibility of making effective use from it?

“[6.1] Given the confines of this consultation, and its remit to focus only on the data policy options for a PDC itself, it would not be appropriate to consult on the whole policy and legislative framework”

Which is to say, you are not soliciting ideas about how to set up a governance regime that will require a nascent PDC to develop structures and processes that seek to innovate in the way public data is collected, processed and exploited, or helps realise a vision where free flowing open public data revitalises the way in which public bodies operate?

***”12.What would be an appropriate timescale for reviewing a PDC or its constituent parts public task(s)?”

Notes

***1. Do the definitions of the key terms go far enough or too far?
“Public services are either provided by public bodies, or providers who have been funded, commissioned or established by statute to provide a service”

I assume the definitions of open data and public services are to be taken together, with the consultation focussing on ‘open (public) data produced by public services’? For such bodies, I assume there is also a formal “data burden” that defines the public data reporting requirements to the centre, as well as devolved data burdens eg into local government from schools? Would it make sense to clarify the notion and extent of data burdens, and the extent to which elements of these (and the organisations they apply to) should be subject to open data requirements? I guess there is also a data burden placed on individual citizens in respect of filing tax forms, for example, that are not subject to openness requirements?

A clear statement at least of data burdens/formal reporting requirements between public bodies that are in scope for mandatory release as open public data should be made available, eg along the lines of http://www.communities.gov.uk/localgovernment/decentralisation/tacklingburdens/singledatalist/ http://getthedata.org/questions/500/data-burden-on-uk-higher-education (I know some work has already been done on this that I used as the basis for a simple data brden visualisation exercise ( http://www.flickr.com/photos/psychemedia/5536836259/ ).)

“Dataset”
It may be useful to distinguish between data collected for operational, administrative or statistical use, as well as the extent to which data produced in the normal course of events is being legitimately requested as is, or whether it must be processed before release (eg http://www.adls.ac.uk/what-is-administrative-data-and-why-use-it-for-research/ http://www.unsiap.or.jp/ms/ms7/DennisP1_OppoChalle.pdf ).

It may also be worth distinguishing between the release of complete data sets, views over the data that represent a query on a complete dataset, and queries, sampling procedures or any other means that are used to generate those data views. For example, providing data relating to performance indicators for a particular school in response to an FOI request from a citizen equates to the provision of a particular view over the database containing perfomance indicators for UK schools as a whole; providing a copy of the database as a whole to a developer of a school comparison website represents the provision of a complete dataset.

Datasets may provide value to others in a variety of ways: for example:
- using complete datasets as the basis of comparison or recommendation services;
- using complete datasets to support statistical analyses, segmentation/clustering of data;
- generating very particular or specific views over the dataset by constructing meaningful and appropriate queries on the datasets. Queries are also reusable, and whilst some cost may be incurred in creating them, making them open, and suitably parameterising them, the marginal cost of reusing the queries is then minimal. It is possible that queries that take a long time to create/optimise become valuable in their own right, and that the dataset and the view can be given away freely. The query unlocks value in the dataset and delivers it to the requester. When it comes to government reporting, where reports include summary views over open datasets, the openness/transparency requirement should not deem to be met unless the query that generates the view from the dataset is also openly published.

Datasets may also include recordsets relating to an individual; where personal access to personal data/mydata is possible, we need to distinguish between the private/personal right for an indvidual to access their data, or an agent acting on their behalf and with their permission, as opposed to general public access.

****2. Where a decision is being taken about whether to make a dataset open, what tests should be applied?
If data is part of a formally defined data burden, should that data burden be tiered in terms of openness requirements, for example along lines of:
- open on submission to the centre;
- open following embargo period and subject to checking by the centre, but with the presumption that it will be opened;
- not open;

Where data is FOIable, that may be taken as evidence in favour of presumed openness. If data is regularly requsted via FOI, it could be made available in open form as a matter of course in order to reduce FOI overheads in the future. When data is released via FOI, it could be made available via an open data site in partial fulfilment of handling the FOI request. When responding to FOI requests for data, the process required to obtain and release that data could be captured and compared with the actual processes relating to operational and administrative use of that data in order to identify whether an open data tap can be introduced into the current data process to open it as a matter of course, or release it efficiently in response to an FOI request.

As the major producer and consumer of public data, public bodies are well placed to benefit from more open public data. “Publicness” and “openness” both help make data accessible for use within and between public bodies, we well as reuse by third parties; accessibility is also improved by timely release of data, and the publication of data using open standards and formats.

Consequences of making data open should also be considered; for example, once released, will there be continued access to regular updates of the data using the same format. (If the data is released sporadically and with inconsistent formats, services that automate the regular collection of the data are not really viable).

****3. If the costs to publish or release data are not judged to represent value for money, to what extent should the requestor be required to pay for public services data, and under what circumstances?

Where work must be done that does not represent value for money (what would an example of this be? Having to get data into a form the public body would never use?), it may be appropriate to consider the amount of value that is added in processing the data that the requester might otherwise be expected to add, for example as just reward for the cost of processing that data. If the raw data is open, and the requester asks for processed data, it may be appropriate to give the raw data away freely but charge for the value add of processing it that the requester seeks to exploit in the course of their business? However, there will also be a tension between people who want to gain access to a small amount of data, either for personal use, research/innovation purposes, and companies who make use of that data in volume as part of a business. In the latter case, we might expect some payment for use of the data once the business is operating, although it could be argued that if the business is profitable, there is a return built in through taxation.

A balance may need to be struck based on the number of independent requests that are likely to be received for a particular data set and the use they wish to put it to. If N requests are made for the data, and all N parties need to do the same work cleaning or processing the data in the same way, that is obviously inefficient. It may be that third parties process and repackage data, for a fee. But the question arises – if data as published is not fit for use by third parties, is it fit for use by the first (producer) or second (‘official consumer’) party, or has the data been produced solely in response to some openness criteria, and not because the data is actually used for anything?

The ability to save cost elsewhere in government may also be an issue. For example, local authorities who make disbursements to care homes need to mitigate against fraud by regularly checking death reports, often through the purchase of commercial death registers or by checking the local newspaper’s death notices. Whilst a cost may be associated with signatories of death certificates ensuring this data enters the public body data chain in an accessible and open way, it may well save costs in multiple other areas of government.

Where data is processed and released in exchange for a payment, would it also be possible for the raw underlying data to also be made available for free so that third parties can, at their own expense, carry out the required processing if they can do so for less overall cost than piecewise purchase of data from the public body?

****4. How do we get the right balance in relation to the range of organisations (providers of public services) our policy proposals apply to? What threshold would be appropriate to determine the range of public services in scope and what key criteria should inform this?

If an organisation is subject to FOI requests, or data it produces and returns as part of an official data burden may be requested through FOI requests, it should be in scope?

Analysis of data processes associated with fulfilling data burden requirements might provide a basis for identifying where in a data process data might reasonably be made public and open.

*****5. What would be appropriate mechanisms to encourage or ensure publication of data by public service providers?
If data related FOI responses are published via open data sites, the open data site can become a repository of commonly requested data and help identify which processes might benefit from releasing open data as a matter of course.

Where public data is reported as a matter of course by the local press and in the local interest, (for example, court reports, planning notices, traffic notices), public bodies might be encouraged to publish the corresponding data in an open way in order to facilitate the local dissemination of that information. Note that much of this data is transitory/may only be relevant for a limited period. In this case, we need to consider: whether there is a public interest in making the data publicly available and open on an archival basis, or not providing archives per se, but responding to requests for archival copies of data; the extent to which third parties can archive/aggregate such data and continue to make it available; whether there are privacy reasons for not supporting archival access (for example, court reports in local newspapers have a “short memory”).

Are there guidelines available that cover the interactions between things like:
- data eligible for release under FOI;
- data that may be redacted on grounds of Data Protection Act
- data covered by Database Right or data that is covered by copyright
- data released through National Statistics ( http://www.legislation.gov.uk/ukpga/2007/18/contents )
- reusable public sector information ( http://www.legislation.gov.uk/uksi/2005/1515/contents/made )

Analysis of data burden reporting process might identify appropriate points at which data can be made open as part of the process. For example, reported data may be posted to an open data site from where it is collected (“pull reporting”). See also: http://blog.ouseful.info/2011/03/18/open-data-processes-taps-query-pathsaudit-trails-and-round-tripping/

And as I responded to the PDC Engagement Exercise, [o]ne particular class of data that interests me is data that is:

1) reported by a local organisation to a central body;
2) using a standardised, templated reporting format,
3) and that is FOIable either from the local organisation, and/or from the central body.

For example, in Higher Education, this might include data on library usage as reported to SCONUL, or marketing information about courses submitted to UCAS.

It can often be hard to find out how to phrase an FOI request to obtain this data as submitted, unless you know the type of reporting form used to submit it.

What I would like to see is the Public Data Corporation acting in part as a Public Data Exchange Directory, showing how different classes of public organisation make standard (public data containing) reports to other public organisations, detailing the standard report formats, with names/identifiers for those forms if appropriate, and describing which sections of the report are FOIable. This could also link in to the list of local council data burdens, for example ( http://www.communities.gov.uk/… and/or the code of practice for local authority transparency ( http://www.communities.gov.uk/… )

The next step would be to introduce a pubsub (publish-subscribe) model in the reporting chain for reporting documents* that are wholly FOIable. This could happen in several ways:

A) /open report publication/ – the publishing organisation could post their report to their opendata reporting store, and the consuming organisation (the one to which the report was being made) would subscribe to that store, collecting the data from there as it was published; third parties could also subscribe to the local publishing store and be alerted to reports as they are published. If co-publication to the central organisation and the public is not appropriate, the report could be witheld from public/press consumption for a specified period of days, or published to the press but not the public under embargo.

B) /open deposit/ – the publishing organisation publishes the report/data to an open deposit box owned by the central organisation which is receiving the report. After a specified period of time, the report is made public (ie published) via that central deposit box.

C) /data corp in the middle/ – a centralised architecture in which local organisations submit public reports to a Public Data Exchange, which then passes them on to the central body to which reports are made, and publishes them to the public, maybe after a fixed period of time.

The intention of all three approaches described above is to provide an open window onto the reporting chain. At the current time, open public data tends to be data that is published via a separate branch “to the public”. In contrast, the above approach suggests that public data publication acts as a view onto all or part of the data as it goes about it’s daily business being published from one organisation to another. That is, public data publication becomes a “tap” onto a dataflow/workflow process.

If one of the desires for data exploitation is to help introduce efficiencies as well as reuse in data related activities, third parties need to be able to work with data as it currently used.

***How will we ensure that Open Data standards are embedded in new ICT contracts?
By providing a test suite as part of the contract that include tests such as running data import/export/query operations against centralised validation services.

***What is the best way to achieve compliance on high and common standards to allow usability and interoperability?
Require data reporting to proceed through open interfaces or interfaces where public data taps can be applied. Released data should be authentic, and representative of data used as part of a public body’s activities or reporting duties rather than data that is produced purely for release on an open data site.

***How would we ensure that public service providers in their day to day decisionmaking honour a commitment to Open Data, while respecting privacy and security considerations.
Take a lead from open source software projects and publish requests via an issue tracker, that can show when an ‘issue’ was raised, what it’s current status is, and how it was resolved. Related approaches include services like WhatDoTheyKnow or GetTheData

***How should public services make use of data inventories? What is the optimal way to develop and operate this?
If we distinguish between datasets, queries on datasets, and reports/data view generated by queries on datsets on the one hand, and data burdens on the other, we can start to map out how queries are used on datasets to generate reports that fulfil data burden requirements. That has the benefit of making the data burden fulfilment process more transparent, as well as contextualising both the way those reports are generated (through exposing the queries) and the original data sets used as a basis for creating reports.

***Should the data that government releases always be of high quality? How do we define quality? To what extent should public service providers “polish” the data they publish, if at all?
One rule of thumb is that the data should be “good enough”. The question then arises, ‘good enough for whom?’. If the data is released and never referred to, its quality is irrelevant as regards the non-existent on-users, although it may signal problems elsewhere. If data is used by a third party and found to contain errors or omissions, the question arises: does the publisher also suffer from those some lack of quality issues (and if so, how are they handling them?); or are they using a different data set as part of the process that the released dataset relates to (and if so, why isn’t that data being released?)

There are different levels of cleanliness we may associate with data: a major issue in many datasets relates to the use of inconsistent labels to refer to the same entity (something that can be addressed by using universal persistent identifiers). Character set encodings can also cause problems, especially where it is hard to identify what character sets are used within a file.

***How should government approach the release of existing data for policy and research purposes: should this be held in a central portal or held on departmental portals?
As I understand the current situation, public body reports often produce summary tables and as part of transparency requirements, release as public data raw datasets that are used to generate those summary tables. In such cases, the query used to generate the summary table from the raw data should also be published. The transparency does not come from releasing summary tables and saying “it summarises that pile of data”. It comes from saying – here is the summary, and here is how it was generated from that data, allowing the observer to check the assumptions of the query, redo the analysis, and so on.

Using services such as Google spreadsheets or Zoho spreadsheets, it is possible to provide a preview view over the data contained in a dataset made available as a simple CSV file (this approach is taken on some datastores). It is also possible to use services such as a Google spreadsheets as a database, and so provide a certain level of intermediate developer access to the raw data as if read access were made available to the database that sourced the released data (eg http://blog.ouseful.info/2010/11/19/government-spending-data-explorer/ ). A range of powerful hosted statistical analysis and visualisation tools are now available that can also provide a user interface layer over over data published in such environments (“analysis at the point of delivery”). For example, the popular R environment can provide an online statistical analysis UI to online hosted datasets via services such as http://www.stat.ucla.edu/~jeroen/ggplot2.html or http://www.rstudio.org/docs/server/getting_started These tools provide an intermediary step that allow interested parties to explore datasets in situ. Recent developments with the Linked Data API ( http://www.epimorphics.com/web/tools/linked-data-api.html ) offer similar capabilities, including the ability to share persistent URLs to queries that are applied to public Linked Data stores such as those hosted under the data.gov.uk umbrella.