OUseful.Info, the blog…

Trying to find useful things to do with emerging technologies in open education

Archive for the ‘Stirring’ Category

Regularly Scheduled FOI Requests as a None Too Subtle Regular OpenData Release Request? And Some Notes on Extending FOI

with 3 comments

A piece of contextualisation in an interview piece with Green MP Caroline Lucas in Saturday’s Guardian (I didn’t do this because I thought it was fun) jumped out at me as I read it: More than 50 energy company employees have been seconded to the government free of charge, and dozens of them to the department of energy and climate change.

Hmm…. /site:gov.uk (secondment OR seconded) DECC/

Secondments google gov.uk

How about the gov.uk site?

gov uk secondment

(I don’t know what’s going in the fight between GDS and the data.gov.uk folk ito getting central gov #opendata info discoverable on the web, but the http://www.gov.uk domain seems to be winning out, not least because for departments who’re in that empire, that’s where any data that eventually linked to from data.gov.uk will actually be published?)

So – it seems folk have been FOIing this sort of information, but it doesn’t look as if this sort of information is being published according to a regular schedule under an #opendata transparency agenda.

Anyone would thing that the UK government wasn’t in favour of a little bit of light being shone on lobbying activity…

(What did happen to the lobbying bill? Oh, I remember, it got through in a form that doesn’t allow for much useful light shedding at all (enactment), and now Labour want to repeal it.)

I guess I could put a request in to the ODUG (Open Data User Group) for this data to be released as open data, but my hunch is it’s not really the sort of thing they’re interested in (I get the feeling they’re not about open data for transparency, but (perhaps unfairly…?!) see them more as a lobbying group (ODUG membership) for companies who can afford to buy data but who would rather the tax payer pays for its collection and the government then gifts it to them).

More direct would be to find a way of automating FOI requests using something like WhatDoTheyKnow that would fire off an FOI request to each central government department once a month asking for that previous months’ list of secondments into and out of that department in the preceding month (or in the month one or two months preceding that month if they need a monthly salary payment cycle or two for that data to become available).

Of course, it does seem a bit unfair that each government department should have to cover the costs of these requests, but as it stands I can’t make an FOI request of companies that choose to engage in this sort of presumably public service.

Making private companies offering public services under contract subject to FOI does seem to be on the agenda again though, after being knocked back around this time last year?:

An extension to the scope of the FOI Act was proposed a few weeks ago in the Public Bill Committee debate of the morning of Tuesday 18 March 2014 on the Criminal Justice & Courts Bill, columns 186-193:

Dan Jarvis: I beg to move amendment 37, in clause 6, page 6, line 29, at end insert—

‘(1A) The Code of Practice must include a requirement that a person carrying out electronic monitoring who is not a public authority as defined by section 3 of the Freedom of Information Act 2000 shall provide information in respect of the carrying out of electronic monitoring in the same manner as if they were such a public authority.’.

The Chair: With this it will be convenient to discuss amendment 38, in schedule 4, page 73, line 25, at end insert—

‘(1A) Where the Secretary of State enters into a contract with another person under paragraph 1(1), and that person is not a public authority for the purposes of section 3 of the Freedom of Information Act 2000, that person shall be designated by the Secretary of State as a public authority for the purposes of that section in relation to that contract.’.

I remind the Committee that this group is about freedom of information provisions as they apply to aspects of the Bill. Members will have the opportunity to debate the detail of secure colleges later.

Dan Jarvis: I serve notice that, unless sufficient assurances are received, we intend to put the amendments to a vote. [ Interruption. ] Dramatic! I sensed for a moment that there was a higher authority raising a concern about these amendments, but I shall plough on regardless, confident in the knowledge that they are true and right.

Anyone who knows the story of Jajo the rabbit will understand what I am about to say. For those members of the Committee who do not know, Jajo was the pet rabbit successfully registered as a court translator and then booked in for shifts following the Ministry of

Column number: 187
Justice’s outsourcing of language service contracts. Jajo’s short-lived translation career says less about his talent and much more about the importance of ensuring that public contracts delivered by private providers are properly managed.
As was touched on, Ministers now have to manage another fall-out. Two private providers of electronic monitoring overcharged the taxpayer by millions of pounds for tagging offenders who had died or moved abroad, or who were already back in prison. That underlines the case for the amendments.

Both amendments would seek to bring non-public providers of public services contracted out under the Bill within the scope of the Freedom of Information Act. Amendment 37 relates to clause 6 and the code of practice that would be issued by the Secretary of State on the processing of data related to electronic monitoring. It would require anyone carrying out monitoring related to the clauses to comply with FOI requests in the same way as public bodies do. Amendment 38 relates to schedule 4 and the arrangements for contracting out secure colleges, which are detailed in part 2. It would require anyone contracted to provide a secure college to comply with freedom of information in the same way. Both our proposals are worthy of consideration by the Committee.

We all know that the landscape of how public services are delivered is changing. The Government spend £187 billion on goods and services with third parties each year, about half of which is estimated to be on contracting out services. About half of all spending on public services now ends up in private providers’ hands and more and more private providers are bidding to take on the responsibility and financial rewards that come with large-scale public contracts. As outsourcing is stepped up, more and more information about public services and public money is being pulled out of the public domain. That presents a particular challenge that we must tackle.

As the Information Commissioner told the Justice Committee last year,

“if more and more services are delivered by alternative providers who are not public authorities, how do we get accountability?”

The rewards that third parties stand to gain need to go hand in hand with the duties of transparency and information sharing. The public should be able to ask about how, and how well, the service they are paying for is being run.

The Freedom of Information Act does provide for supply-chain companies to be considered to be holding information on behalf of a public authority. In practice, however, contracted providers in the justice sector are not subject to anywhere near the same transparency requirements as publicly-run services. Private prisons, for example, are not subject to FOI in the same way as public prisons and the experience of G4S, Serco and others will have influenced many other companies not to be as forthcoming as they might have been. That is why we need to build freedom of information into the contracts that the Government make with third parties.

The Committee will be aware that such an approach was recommended by the Public Accounts Committee in its excellent report published last week. It made the

Column number: 188
point that many Departments are not providing information on how those contracts work on the grounds of commercial confidentiality. The public will not accept that excuse for much longer.
Let me conclude my remarks by offering the Committee a final quote. Someone once said:

“Information is power. It lets people hold the powerful to account”

and it should be used by them to hold their

“public services to account”.

I agree with the Prime Minister. Two years ago, he spoke about

“the power of transparency”

and

“why we need more of it.”

He also spoke of leading

“the most transparent Government ever.”

Labour has pledged that the next Labour Government will deal with the issue by bringing companies providing public contracts into the scope of FOI legislation.

Freedom of information can be uncomfortable. It can shed light on difficult issues and be problematic for Government Ministers, but that is the point. The Committee has the opportunity today to improve the Bill and to get a head start.

Dr Huppert: I will not detain the Committee. I share the concern about the lack of FOI for private organisations providing public services. My colleagues and I have expressed concerns about that for many years, and the previous Government were not very good at accepting that. It is good news that the Labour party may undo that error.

Mr Slaughter: Can the hon. Gentleman say what steps he and the coalition have taken to extend FOI in the past four years?

Dr Huppert: Not as many as I would like, but we have seen progress in some areas; we did not see any at all when the hon. Gentleman was a Minister. I hope we will see the correct drive. I share the concern that we need transparency when public services are delivered by private companies. They must not be shielded. I look forward to hearing what the Minister has to say, because he has commented on such issues before.

It is important that the matter should be dealt with on a global scale. I think the shadow Minister would agree that the case is broader. I hope to hear from the Minister that there will be more work to look at how the issue can be addressed more generally, rather than just in a specific case. That would probably require amendment of the Freedom of Information Act. That is probably the best way to resolve the issue, rather than tacking it on to this area, but I absolutely share the concerns. I hope we can see more transparency, both from the Government—we are seeing that—and from the private sector as it performs public functions.

Yasmin Qureshi: The Justice Committee, of which I am a member, looked into the Freedom of Information Act and how it has been operating since it was passed many years ago. We spoke to different groups of people,

Column number: 189
representatives of councils and local authorities, the Information Commissioner and pressure groups. Generally, the view was that the Freedom of Information Act has been a force for good. The thing that people mentioned time and again was the fact that it applies only to public authorities and has a narrow remit in private companies. A lot of concern was expressed about that.
As my hon. Friend the Member for Barnsley Central said, just under £200 billion is being spent by the Government for private companies to carry out public work. The number of outsourcings could increase, especially in the criminal justice system. In the probation service there will be contracting out and privatisation, as well as changes in the criminal justice system in relation to legal aid and suchlike. We have concerns about the criminal justice system and the number of companies that will be carrying out work that the state normally does. It is an important issue.

Will the Minister give us an undertaking for whenever Government money is given to carry out work on behalf of the Government? Local authorities and Government Departments have to provide information, and it should be the same for private companies. At the moment, as the shadow Minister mentioned, the agencies providing some of the public work give some information, but it is not enough.

It is often hard to get information from private companies. It is important for the country that we know where public money is being spent and how private companies respond to such things. We can have party political banter, but freedom of information was introduced many years ago and has been working well. Freedom of information needs to be extended in light of the new circumstances. I ask for a clear commitment from the Government that they will encapsulate that in the Bill. They now have that opportunity; the Labour party has said that, if it was in government, it would certainly do so. The lacunae and the gaps would be addressed by the amendment, which would make it clear exactly how the regime applies. [Interruption.]

10.30 am
The Chair: I apologise for the background noise. We are looking into the cause.

Jeremy Wright: Thank you, Mr Crausby. I hope Jajo the rabbit is not responsible.

As the hon. Member for Barnsley Central said, amendment 37 seeks to introduce a requirement as to the contents of the code of practice that the Secretary of State will issue under proposed new section 62B of the Criminal Justice and Court Services Act 2000, which is to be introduced through clause 6. The Secretary of State would have to include provisions in the code of practice requiring providers of outsourced electronic monitoring services to make information available in the same manner as if they were subject to the provisions of the Freedom of Information Act. The aim of the amendment seems essentially to extend the Act to providers of electronic monitoring not already subject to its provisions.

Amendment 38 has the same basic intention in that it seeks to extend the Freedom of Information Act to providers of secure colleges that have entered a contract with the Secretary of State to do so under schedule 4. The approach differs, however, because amendment 38

Column number: 190
would extend the Act directly, whereas amendment 37 seeks to extend its obligations through code of practice guidance.
In other words, both amendments would require private providers not currently subject to the Freedom of Information Act to make information available both in response to FOI requests and proactively through publication schemes. Section 5 of the Act already provides a power to extend the Act’s provisions to contractors providing public services. For reasons I will try to outline, the Government do not currently propose to adopt that approach and are adopting an alternative method to ensure transparency. I am aware, however, of the long-standing and serious concerns raised on the position under the Act of private providers of public services. It might help the hon. Member for Hammersmith to know that the Government are committed to, and have taken steps to extend, the Act. More than 100 additional organisations have been included since 2010, and we are considering other ways in which its scope may be widened.

The issue of outsourced public services was considered in some detail during post-legislative scrutiny of the Freedom of Information Act by the Select Committee on Justice in 2012. I do not know whether the hon. Member for Bolton South East was a member of the Committee of that time, but the Committee rightly issued a reminder that

“the right to access information is crucial to ensuring accountability and transparency for the spending of taxpayers’ money”.

The Committee recommended the use of contractual provisions, rather than the formal extension of the Act, to ensure that transparency and accountability are maintained. In particular, the Committee said:

“We believe that contracts provide a more practical basis for applying…outsourced services than partial designation of commercial companies under section 5 of the Act”.

The Committee also feels that

“the use of contractual terms to protect the right to access information is currently working relatively well.”

The Government’s approach is consistent with that recommended by the Justice Committee.

In addition to information being made available proactively, the Government are taking steps to issue a revised code of practice under section 45 of the Freedom of Information Act to promote transparency on outsourced public services in response to FOI requests. The code of practice will be issued later this year and will promote and encourage the use and enforcement of contractual obligations to ensure that private bodies not subject to the Act provide appropriate assistance where information about outsourced public services is requested from bodies that are subject to the Act.

The Government recognise that only a small amount of information held by private providers is currently often formally subject to the Act. Our code of practice will encourage public authorities to go further, to interpret their freedom of information obligation broadly and to release more information on a voluntary basis, where it would be in the public interest to do so. In the event of non-compliance, it will also be possible for the Information Commissioner to issue and publish a practice recommendation setting out steps that, in his view, the public authority should take to promote conformity with the guidance.

Column number: 191
Mr Slaughter: I seem to remember taking part in the Westminster Hall debate arising out of the Justice Committee’s deliberation and I do not think that it was very happy with the approach that the Government are taking, particularly where they are seeking to restrict freedom of information further. Does the hon. Gentleman accept on the basis of what he has just said that this will not be a level playing field and that the same requirements that apply to public bodies will not apply to private organisations undertaking an effectively identical role? Does he accept that, whatever the merits of his scheme, it does not to far enough and does not address the comments of my hon. Friend the Member for Barnsley Central?

Jeremy Wright: The hon. Gentleman will recognise that the organisations we are talking about extending the provisions of the Act to cover vary hugely in size and level of resources. The concern is to draw the appropriate balance between giving correct access to information and not imposing intolerable burdens on organisations, particularly smaller ones. That is the balance that has to be struck. We are looking at ways in which we can continue to make public authorities responsible for supplying information but ensure that it comes from the place where it originated, which may be those other organisations.

Mr Slaughter: That is a different argument and one that is often tried. It was tried in relation to universities and to the smaller district councils much beloved of the hon. Member for Bromley and Chislehurst. There are already limitations within the Act. There are safeguards for organisations in terms of the amount of time and cost. Why are they not sufficient?

Jeremy Wright: As I said, there is a balance to be struck. We attempt to strike that balance correctly with our proposals. If I can explain what we want to do a little more fully, perhaps the hon. Gentleman will be reassured—although frankly I doubt it. There is an opportunity for us to look at the issue in a sensible way with the code of practice. Applying our forthcoming code of practice guidance across the public sector will ensure that transparency and response to freedom of information requests will be maintained in a consistent way. This is preferable—I agree with my hon. Friend the Member for Cambridge—to the more piecemeal approach promoted by amendments 37 and 38.

The success of our own code of practice will be monitored by the Ministry of Justice and the Information Commissioner. We were clear in our response to post-legislative scrutiny of the Freedom of Information Act that, should this approach yield insufficient dividends, we will consider what other steps are necessary. In summary, we are committed to ensuring transparency in relation to all outsourced public services, including electronic monitoring and secure colleges. We are taking steps to ensure that through the code of practice to be issued later this year. On that basis, I invite the hon. Gentleman to withdraw his amendment.

Yasmin Qureshi: The Minister referred to the Select Committee on Justice and its recommendations. As you know, without going into the detail of that discussion, Select Committee recommendations sometimes tend to

Column number: 192
be compromises. At the time, three issues were in the mind of the Select Committee. First, it did not realise that a legislative opportunity would come so soon in which to put the measure in a more codified way with a clearer legal obligation. Secondly, there was quite a lot of discussion about private companies.
The Select Committee accepted that the Freedom of Information Act should not apply to purely private companies carrying out purely private work; it was not really arguing against that. However, here we have an opportunity to codify once and for all in legislation the provision that the FOIA should apply whenever public money is paid to a private company to carry out work. That would be a fairly straightforward provision. I do not see why we need to go down the complicated route of using a code of practice, putting in a specific provision in a new contract each time something happens. Why can we not just have a general provision that applies to every situation?

Jeremy Wright: I was a member of the Justice Committee before the hon. Lady was, so I understand her point that recommendations of the Select Committee are a matter of discussion and compromise. However, they are made on a cross-party basis, and paid all the more attention to for that reason. I quoted directly from the Select Committee’s conclusions in what I said earlier.

On the hon. Lady’s other point, this may be an earlier legislative opportunity than the Select Committee anticipated, but of course, it is only an opportunity in relation to specific policies. Again, I rather agree with the point made earlier by my hon. Friend the Member for Cambridge: there is an argument for addressing the issue, not on a piecemeal basis, but more comprehensively.

The hon. Lady’s final point is that the approach that we have set out—using a code of practice—is inadequate and that a statutory approach should be introduced by amending primary legislation. An initial approach of using a code of practice is a sensible one. She will recognise that amendment 37, tabled by the hon. Member for Barnsley Central, deals with a requirement in a code of practice, not primary legislation. Amendment 38 is different, but in relation to electronic monitoring, on which a number of concerns have been expressed, the hon. Gentleman’s chosen vehicle is a code of practice. The code of practice approach appears to be welcomed by both sides of the Committee.

Dan Jarvis: I have listened carefully to the Minister’s response. Clearly, we will want to look carefully at the detail of what he has said about a code of practice.

I agree with my hon. Friend the Member for Bolton South-East that the Committee has an opportunity this morning to make progress on redefining the freedom of information. I have heard the Minister’s response to that point, but the reality is that the move would be popular with the public.

There is no doubt that the landscape in which public services are delivered is changing. The Opposition have pledged to reform freedom of information if we are in government from 2015. I am mindful of the Prime Minister’s comments, which I quoted earlier. He said:

Column number: 193
“Information is power. It lets people hold the powerful to account”,

and it should be used by them to hold their public services to account.

Mike Kane: Does my hon. Friend agree that, as the contracting out of public services expands, the public’s right to information shrinks?

Dan Jarvis: I agree absolutely. There is a degree of inevitability that we will see change in the area. The debate is about how we do it, and it is important that we have that debate. We have tabled the amendments partly so that we can take the opportunity to debate such issues.

Mr Slaughter: There is another point here, which is that the Ministry of Justice is particularly vulnerable on the issue. We have had the privatisation of the probation service and the scandals regarding tagging. We will come to later in the Bill to proposals about the externalisation of the collection of fines and other matters. First, that is going on wholesale in the Department, and secondly, it is defective in many aspects. It is particularly relevant that the Minister should accept that the proposals in the Bill are not sufficient.

Dan Jarvis: My hon. Friend is right. In the context of the delivery of public services within the Ministry of Justice remit, this is a particularly relevant, timely and important issue. It has been incredibly useful to have the opportunity to debate it owing to the tabling of the amendments.

10.45 am
I mentioned that I was mindful of the Prime Minister’s comments, and I am mindful of the fact that the Justice Secretary has also indicated a desire to reform freedom of information. Given that there is a general acknowledgment that the status quo is not acceptable and despite what the Minister has said in response to our amendment, I will press it to a vote.

The amendment was defeated.

An hour or so later, the government took this line:

Daily Hansard, Commons, Tuesday 18 March 2014 – c.639

Freedom of Information Act
23. Lindsay Roy (Glenrothes) (Lab): What plans he has to bring forward legislative proposals to expand the scope of the Freedom of Information Act 2000.

The Minister of State, Ministry of Justice (Simon Hughes): There has been good progress in extending the implementation of the Freedom of Information Act because the coalition Government pledged to extend its scope to provide greater transparency. We extended it in 2010 to academies, in 2011 to the Association of Chief Police Officers, the Financial Ombudsman Service and the Universities and Colleges Admissions Service, and last year to 100 companies wholly owned by more than one public authority. The next item on the agenda is to do with Network Rail, and we are awaiting a view from the Department for Transport as to whether it thinks it would be appropriate for that to be implemented this year.

Lindsay Roy: What benefits have accrued to the Government and citizens from the implementation of the Act, and when does the Minister plan to extend its scope further?

Simon Hughes: We intend to extend it further as soon as is practical. One specific issue that I hope will be of interest to the hon. Gentleman—as it is to colleagues of his, including those who have come to see me about it—is that we intend to publish a revised code of practice to make sure that private companies that carry out public functions have freedom of information requirements in their contracts and go further than that. We hope that that will be in place by the end of this year.

Mr Mark Harper (Forest of Dean) (Con): There is one area where the Minister should perhaps look at narrowing the scope of the Act, because my understanding is that requests can be made by anybody anywhere on the face of the earth; they do not have to be British citizens. It is not the role of the British Government to be a taxpayer-funded research service for anyone on the globe. May I suggest that he narrow the scope to those for whom the Government work—citizens of our country?

Simon Hughes: I well understand my hon. Friend’s point. There will be two consultations this year: first, on precisely such issues about the scope of the current legislation to make sure that it is not abused while we retain freedom of information as a principle of Government; and secondly, on extending it to other areas where we have not gone so far.

Dr Huppert:I read out the quote from someone who has made the position clear when it comes to private companies carrying out public functions. Indeed, the code of practice has exactly the wording used in amendment 11, which the hon. Gentleman supported when we debated it on Tuesday. I do not want to take up too much of the Chairman’s kindness to discuss an issue that was rejected at that point, but it is happening as we wanted.

The matter was also touched upon a couple of days later in a Public Bill Committee on the Criminal Justice and Courts Bill (Official Report, Thursday 20 March 2014, 257-259) where accountability around public contracts delivered by private provides was being discussed:

Mr Slaughter: Absolutely not. I hope that the hon. Gentleman has read the article about Jago the rabbit that my hon. Friend the Member for Barnsley Central (Dan Jarvis) and I wrote for The Independent yesterday [It’s time we extended Freedom of Information to public services run by private companies – just ask Jago the Rabbit], which dealt with what should be done, which is to bring these companies within the ambit of FOI, and what the Minister of State did—with his usual skill, shall we say?—at Justice questions on Tuesday. He implied that that was what was going to happen, whereas in fact he was doing nothing more than putting round the line that the Cabinet Office has already indicated.

If I am wrong about that, I will give way in a moment and the hon. Gentleman can come back to me, but my understanding is that the Government—both parts of it, as long as they are just about coalescing—are of the view that the contracts that are drawn up should include this notional transparency. That is to say that they will encourage public authorities to encourage private companies to put clauses into contracts that will expose as much as possible, within the realms of commercial confidentiality. So the contracts will be open, with publication on websites and so forth of as much information about the contract as the two parties think fit. What we will not have is a duty on those private companies—in so far as they are carrying out public functions—to comply with the terms of the Freedom of Information Act, as would be the case in the public sector.

I accept that they are two sides of the same coin. On the one hand, of course it is a good idea that the information is made available voluntarily, but if it is not—either because the company does not choose to do so or because the contract is not drafted sufficiently well to ensure that it must—the citizen must have the right, through FOI, to require that information to be made available. As far as I am concerned, that is not what was said on Tuesday. I know that there is consultation going on, but if it is the intention of the Government—at least the Liberal Democrat part of the Government—to follow the line taken by my right hon. Friend the Member for Tooting (Sadiq Khan), the shadow Lord Chancellor, which he has repeated often in recent months, and require all those private companies performing public functions to come within the requirements of the Freedom of the Information Act, I would be pleased if the hon. Gentleman said so now.

Mr Slaughter:I take from that comment that even the hon. Gentleman does not understand what the Minister of State, Ministry of Justice, the right hon. Member for Bermondsey and Old Southwark, says, so opaque is it. If nobody, including the Minister, is going to answer my question, the answer will no doubt come out in the wash on a later occasion. However, it seems to me that that is not what is being promised. If it were, the Minister would be jumping up and claiming credit for it, but he is not. Therefore, I assume that that is not the case.

The significance of that is that those four companies about which I have just raised doubts—G4S, Serco, Capita, and we can safely add Atos—all told the Public Accounts Committee that they were prepared to accept the measures that the Committee proposed. It therefore appears that the main barrier to greater transparency lies within Government.

That is where we are. Even the companies that want to put themselves and the interests of their shareholders first are more keen on transparency and on answering the legitimate questions that are being asked by everyone— from ourselves to the chief inspector of prisons—than this Government are.

I say that because if we are to take this further leap down that path, it is only right that the Government do not just challenge, as the Minister has said, acknowledged frauds, but look at the entire performance behaviour, as well as the number of available companies that could step into the breach and deal with these matters.

What we must conclude from the conjunction of clauses 17 and 18 is that, first, the Government are prepared to take this leap in the dark, in terms of the reconfiguration of the youth estate and, secondly, that they are prepared to leave that entirely in the hands of the people who failed so many times in so many contracts, not least in running parts of the adult prison service.

For more on some of the specifics, see the House of Commons Public Accounts Committee report on “Contracting out public services to the private sector”, which for example recommended “that the Cabinet Office should explore how the FOI regime could be extended to cover contracts with private providers, including the scope for an FOI provision to be included in standard contract terms; that neither the Cabinet Office nor departments should routinely use commercial confidentiality as a reason for withholding information about contracts with private providers; [and that] The Cabinet Office should set out a plan for departments to publish routinely standard information on their contracts with private providers”.

There’s also a couple of related private members bills floating around at the moment – Grahame Morris’ empty Freedom of Information (Private Healthcare Companies) Bill 2013-14 “to amend the Freedom of Information Act 2000 to apply to private healthcare companies”, and Caroline Lucas’ Public Services (Ownership and User Involvement) Bill 2013-14 “to put in place mechanisms to increase the accountability, transparency and public control of public services, including those operated by private companies”. The latter >a href=”http://www.publications.parliament.uk/pa/bills/cbill/2013-2014/0160/cbill_2013-20140160_en_2.htm#l1g5″>proposes:

5 Transparency
(1) Where a relevant authority starts the process of procurement for a public services contract, it must make available to the public details of all bids received prior to the conclusion of the procurement exercise.
(2) Where a relevant authority enters into a public services contract, details of that contract shall be made available to the public within 28 days of the procurement decision.

6 Freedom of information
(1) The Secretary of State must designate as a public authority, pursuant to section 5(1)(b) of the Freedom of Information Act 2000, companies or other bodies which enter into a public services contract.
(2) “Public services contract” has the meaning contained within section 8 of this Act.
(3) The Secretary of State shall maintain a list of companies designated under section 6(1) of this Act.
(4) Requests under the Freedom of Information Act 2000 in respect of such companies or bodies can only be made in respect of information relevant to the provision of a public services contract.
(5) The Secretary of State must designate as a public authority, pursuant to section 5(1)(b) of the Freedom of Information Act 2000, any utility company subject to regulation by regulatory authorities as defined in section 8.

Finally, on the accountability and transparency thing, there’s a consultation on at the moment regrading “smaller authorities with an annual turnover not exceeding £25,000, including parish councils, [who] will be exempt from routine external audit” but instead will be subject to a transparency code (Draft transparency code for parish councils – consultation).

Related: Spending & Receipts Transparency as a Consequence of Accepting Public Money? If you accept public money for contracts that would otherwise be provided by a public service you should be subject to the same levels of FOI and transparency reporting. Why should public services have to factor this in to their bids for running a service when private companies don’t?

Other reading to catch up on: Commons Public Administration Select Committee [PASC] Report on Statistics and Open Data (evidence).

Written by Tony Hirst

April 6, 2014 at 10:25 am

Posted in opengov, Policy, Stirring

Tagged with , ,

MOOC Busting: Personal Googalytics…

Reading Game Analytics: Maximizing the Value of Player Data earlier this morning (which I suggest might be a handy read if you’re embarking on a learning analytics project…) I was struck by the mention of “player dossiers”. A Game Studies article from 2011 by Ben Medler- Player Dossiers: Analyzing Gameplay Data as a Reward describes them as follows:

Recording player gameplay data has become a prevalent feature in many games and platform systems. Players are now able to track their achievements, analyze their past gameplay behaviour and share their data with their gaming friends. A common system that gives players these abilities is known as a player dossier, a data-driven reporting tool comprised of a player’s gameplay data. Player dossiers presents a player’s past gameplay by using statistical and visualization methods while offering ways for players to connect to one another using online social networking features.

Which is to say – you can grab your own performance and achievement data and then play with it, maybe in part to help you game the game.

The Game Analytics book also mentioned the availability of third party services built on top of game APIs that let third parties build analytics tools for users that are not otherwise supported by the game publishers.

Hmmm…

What I started to wonder was – are there any services out there that allow you aggregate dossier material from different games to provide a more rounded picture of your performance as a gamer, or maybe services that homologate dossiers from different games to give overall rankings?

In the learning analytics space, this might correspond to getting your data back from a MOOC provider, for example, and giving it to a third party to analyse. As a user of MOOC platform, I doubt that you’ll be allowed to see much of the raw data that’s being collected about you; I’m also wary that institutions that sign up to MOOC platforms will also get screwed by the platform providers when it comes to asking for copies of the data. (I suggest folk signing their institutions up to MOOC platforms talk to their library colleagues, and ask how easy it is for them to get data, (metadata, transaction data, usage data etc etc) out of the library system vendors, and what sort of contracts got them into the mess they may admit to being in.)

(By the by, again the Game Analytics book made a useful distinction – that of viewing folk as customers, (i.e. people you can eventually get money from), or as players of the game (or maybe in MOOC land, learners). Whilst you may think of yourself as a player (learner), what they really want to do is develop you as a customer. In this respect, I think one of the great benefits of the arrival of MOOCs is that it allows us to see just how we can “monetise” education and let’s us talk freely and, erm, openly, in cold hard terms about the revenue potential of these things, and how they can be used as part of a money making/sales venture, without having to pretend to talk about educational benefits, which we’d probably feel obliged to do if we were talking about universities. Just like game publishers create product (games) to make money, MOOCspace is about businesses making money from education. (If it isn’t, why is venture capital interested?))

Anyway, all that’s all by the by, not just the by the by bit: this was just supposed to be a quick post, rather than a rant, about how we might do a little bit to open up part of the learning analytics data collection process to the community. (The technique generalises to other sectors…) The idea is built on appropriating a technology that many website publishers use to collect data, the third party service that is Google Analytics (eg from 2012, 88% of Universities UK members use Google Analytics on their public websites). I’m not sure how many universities use Google Analytics to track VLE activity though? Or how many MOOC operators use Google Analytics to track activity on course related pages? But if there are some, I think we can grab that data and pop it into a communal data pool; or grab that data into our own Google Account.

So how might we do that?

Almost seven years ago now – SEVEN YEARS! – in a post entitled They Stole OUr Learning Environment – Now We’re Stealing It Back, I described a recipe for customising a VLE (virtual learning environment – the thing that MOOC operators are reimagining and will presumably start (re)selling back to educational institutions as “Cloud based solutions”) – by injecting a panel that allowed you to add your own widgets from third part providers. The technique relied on a browser extension that allowed you to write your own custom javascript programmes that would be injected into the page just before it finished loading. In short, it used an extension that essentially allowed you to create you own additional extensions within it. It was an easy way of writing browser extensions.

That’s all a rather roundabout way of saying we can quite easily write extensions that change the behaviour of a web page. (Hmm… can we do this for mobile devices?) So what I propose – though I don’t have time to try it and test it right now (the rant used up the spare time I had!) – is an extension that simply replaces the Google Analytics tracking code with another tracking code:

- either a “common” one, that pools data from multiple individuals into the same Google Analytics account;
- or a “personal” one, that lets you collect all the data that the course provider was using Google Analytics to collect about you.

(Ideally the rewrite would take place before the tracking script is loaded? Or we’d have to reload the script with the new code if the rewrite happens too late? I’m not sure how the injection/replacement of the original tracking code with the new one actual takes place when the extension loads?)

Another “advantage” of this approach is that you hijack the Google Analytics data so it doesn’t get sent to the account of the person whose site you’re visiting. (Google Analytics docs suggest that using multiple tracking codes is “not supported”, though this doesn’t mean it can’t be done if you wanted to just overload the data collection (i.e. let the publisher collect the data to their account, and you just grab a copy of it too…).

(An alternative, cruder, approach might be to create an extension that purges Google Analytics code within a page, and then inject your own Google Analytics scripts/code. This would have the downside of not incorporating the instrumentation that the original page publisher added to the page. Hmm.. seems I looked at this way back when too… Collecting Third Party Website Statistics (like Yahoo’s) with Google Analytics.)

All good fun, eh? And for folk operating cMOOCs, maybe this represents a way of tracking user activity across multiple sites (though to mollify ethical considerations, tracking/analytics code should probably only be injected onto whitelisted course related domains, or users presented with a “track my activity on this site” button…?)

Written by Tony Hirst

August 28, 2013 at 3:01 pm

Posted in Stirring, Thinkses

Tagged with

Fragmentary Observations from the Outside About How FutureLearn’s Developing

I’m outside the loop on all matters FutureLearn related, so I’m interested to see what I can pick up from fragments that do make it onto the web.

So for example, from a presentation by Hugh Davis to the M25 Libraries conference April 2013 about Southampton’s involvement with FutureLearn, Collaboration, MOOCs and Futurelearn, we can learn a little bit about the FutureLearn pitch to partners:

FutureLEarn Overview

More interesting, I think, is this description of what some of the FutureLearn MOOCs might look like:

MOOC Structure

“miniMOOCs” containing 2 to 3 learning units, each 2-6 hours of study time, broken into 2-3 self-contained learning blocks (which suggests 1-2 hours per block).

So I wonder, based on the learning block sequence diagram, and the following learning design elements slide:

learning design

Will the platform be encouraging a learning design approach, with typed sequences of blocks that offer templated guides as to how to structure that sort of design element? Or is that way off the mark. (Given the platform is currently being built, (using Go Free Range for at least some of the development, I believe), it’s tricky to see how this is being played out, given courses and platform both need to ready at the same time, and it’s hard to write courses using platform primitives if the platform isn’t ready yet?)

Looking elsewhere (or at least, via @patlockley), we may be able to get a few more clues about the line partners are taking towards FutureLearn course development:

futurelearn job ad - LEeds

Hmm, I wonder – would it be worth subscribing to jobs feeds from the partner universities over the next few months to see whether any other FutureLearn related posts are being opened up? And does this also provide an opportunity for the currently rather sparse FutureLearn website to start promoting those jobs ads? And come to that, how come the jobs that have been appointed at FutureLearn weren’t advertised on the FutureLearn website…?

Because jobs have been appointed, as LinkedIn suggests… Here’s who’s declaring an association with the company at the moment:

futurelearn on linkedIN

We can also do a slightly broader search:

futurelearn search

There’s also a recently closed job ad with a role that doesn’t yet appear on anyone’s byline:

global digital marketing sstrategist

So what roles have been filled according to this source?

  • CEO
  • Head of Content
  • Head of UK Education & HE Partnerships
  • CTO
  • Senior Project Manager / Scrum Master (Contract)
  • Agile Digital Project Manager
  • Product manager
  • Marketing and Communications Assistant
  • Interim HR Consultant
  • Learning Technologist
  • Commercial and Operations Director for Launch
  • Global Digital Marketing Strategist

Here’s another one, Academic Lead [src].

By the by, I also notice that the OU VC, Martin Bean, has just been appointed as a director of FutureLearn Ltd.

Exciting times, eh…?!;-)

Related: OU Launches FutureLearn Ltd

PS v loosely related (?!) – (Draft) Coursera data export policy

PPS I also noticed this the other day – OpenupEd (press release) an EADTU co-ordinated portal that looks like a clearing house for OER powered MOOCs from universities across the EU (particularly open universities, including, I think, The OU…;-)

Written by Tony Hirst

April 25, 2013 at 9:46 am

Posted in Stirring

Tagged with

Moving Machines…

I’ve just taken on a new desktop computer – the first desktop machine I’ll have used as daily machine for seven or eight years. As with every new toy, there is the danger of immediately filling it with the same crud that I’ve got on my current laptop, but I’m going to try to limit myself to installing things that I actually use…

My initial download list (the computer is a Mac):

  • A lot of files I work with are on Google docs, so I don’t actually need to install them at all – I just need a browser to access them
  • an alternative browser: Macs come with Safari preinstalled but I tend to use Chrome; I don’t sign in to Chrome, although I do use it on several machines. Being able to synch bookmarks would be handy, but I’m not sure I want to inflict the scores of open tabs I have onto every browser I open…
  • Dropbox desktop: I need to rethink my Dropbox strategy, and indeed the way I organise files, but Dropbox on the desktop is really handy…having downloaded and configured the client, it started synching my Dropbox files by itself (of course…;-). I’ll probably add the Google Drive dektop client at some point too, but in that case I definitely need a better file management strategy…
  • Gephi: for playing with network visualisations, and one of the main reasons for getting the new machine. As Gephi is a Jave app, I also needed to download a Java runtime in order to be able to run it
  • Rstudio: I considered not bothering with this, pondering whether I could move wholesale to the hosted RStudio at crunch.kmi.open.ac.uk, but then went with the desktop version for several reasons: a) I tinker with RStudio all the time, and don’t necessarily want to share everything on Crunch (not because users can see each others’ files even if they aren’t public, rather: there’s the risk Crunch may disappear/become unavailable/I might be cast out of the OU etc etc); b) the desktop version plays nicely with git/github…
  • Git and Git for Mac: I originally downloaded Git for Mac, a rather handy UI client, thinking it would pull down a version of Git for the commandline that RStudio could play with. It didn’t seeem to, so I pulled a git installer down too;
  • Having got Git in place, I cloned one project I’m currently working on from Github using RStudio, and another using Git for Mac; the RStudio project had quite a few package dependencies (ggplot2, twitteR, igraph, googleVis, knitr) so I installed them by hand. I really need to refactor my R code so that it installs any required packages if they haven’t already been installed.
  • One of the things I pulled from Github is a Python project; it has a few dependencies (simplejson (which I need to update away from?), tweepy, networkx, YQL), so I grabbed them too (using easy_install).
  • For my Python scribbles, I needed a text editor. I use TextWrangler on my laptop, and saw no reason to move away from it, so I grabbed that too. (I really need to become a more powerful user of TextWrangler – I don’t really know how to make proper use of it at all…)
  • Another reason for the big screen/bigger machine was to start working with SVG files – so I grabbed a copy of Inkscape and had a quick play with it. It’s been a long time since I used a mouse, and the Mac magic mouse seems to have a mind of its own (I far prefer two-finger click to RSI inducing right-click but haven’t worked out how/if magic mouse supports that?) but I’ve slowly started to find my way round it. Trying to import .eps files, I also found I needed to download and install Ghostscript (which required a little digging around until I found someone who’d built a Mac package/installer…)
  • I am reluctant to install a Twitter client – I think I shall keep the laptop open and running social tools so as not to distract myself by social conversation tools on the other machine…
  • I guess I’ll need to install a VPN client when I need to login to the OU VPN network…
  • I had a brief go at wiring up Mac mail and iCal to the OU’s Outlook client using a Faculty cribsheet, but after a couple of attempts I couldn’t get it to take so guess I’ll just stick with the Outlook Web App.

PS One of the reasons for grabbing this current snapshot of my daily tools is because the OU IT powers that be are currently looking at installing OU standard desktops that are intended to largely limit the installation of software to software from an approved list (and presumably offer downloads from an approved repository). I can see this has advantages for management, (and might also have simplified my migration?) but it is also highly restrictive. One of the problems with instituting too much process is that folk find workarounds (like acquiring admin passwords, rather than being given their own admin/root accounts from the outset) or resetting machines to factory defaults to get around pre-installed admin bottlenecks. I appreciate this may go against the Computing Code of Conduct, but I rarely connect my machines directly to the OU network, instead favouring eduroam when on campus (better port access!) and using VPN if I ever need access to OU network services. Software is the stuff that allows computers to take on the form of an infinite number of tools – the IT stance seems to take the view that it’s a limited purpose tool and they’re the ones who set the limits. Which makes me wonder: maybe this is just another front on the “Coming Civil War over General-purpose Computing”…?

Written by Tony Hirst

September 13, 2012 at 8:41 pm

Posted in Admin..., Stirring

Enter the Market – Course Data

I’m not at Dev8Ed this week, though I probably should be, but here’s what I’d have probably tinkered with had I gone – a recipe for creating a class of XCRI (course marketing data) powered websites to support course choice on a variety of themes and that could be used to ruthlessly and shamelessly exploit any and every opportunity for segmenting audiences and fragmenting different parts of the market for highly targeted marketing campaigns. So for example:

  • let’s start with something easy and obvious: russelgroupunis.com (sic;-), maybe? Search for courses from Russell Group (research intensive) universities on a conservatively branded site, lots of links to research inspired resources, pre-emptively posted reading lists (with Amazon affiliate codes attached); then bring in a little competition, and set this site up as a Waitrose to the Sainsburys of 1994andallthat.com, a course choice site based around the 1994 Group Universities (hmmm: seems like some of the 1994 Group members are deserting and heading off to join the Russell Group?); worthamillionplus.com takes the Tesco ads for the Million+ group, maybe, and unireliance.com (University Alliance) the Morrisons(?) traffic. (I have no idea if these uni group-supermarket mappings work? What would similarly tongue-in-cheek broadsheet/tabloid mappings be I wonder?!). If creative arts are more your thing, there could be artswayforward.com for the UKIAD folk, perhaps?
  • there are other ways of segmenting the market, of course. University groupings organise universities from the inside, looking out, but how about groupings based on consumers looking in? At fiveAgrades.com, you know where the barrier is set, as you do with 9kQuality.com, whereas cheapestunifees.com could be good for bottom of the market SEO. wetakeanyone.com could help at clearing time (courses could be identified by looking at grade mappings in course data feeds), as could the slightly more upmarket universityclearingcourses.com. And so on
  • National Student Survey data could also play a part in automatically partitioning universities into different verticals, maybe in support of FTSE-30 like regimes where only courses from universities in the top 30 according to some ranking scheme or other are included. NSS data could also power rankings of course. (Hmm… did I start to explore this for Course Detective? I don’t remember…Hmmm…)

The intention would be to find a way of aggregating course data from different universities onto a common platform, and then to explore ways of generating a range of sites, with different branding, and targeted at different markets, using different views over the same aggregated data set but similar mechanics to drive the sites.

PS For a little inspiration about building course comparison websites based around XCRI data, NSS data and KIS data, it may be worth looking at how the NHS does it (another UK institution that’s hurtling towards privatisation…): for example, check out NHS Choices hospitals near you service, or alternatively compare GPs.

PPS If anyone did start to build out a rash of different course comparison sites on a commercial basis, you can bet that as well as seeking affiliate fees for things like lead generation (prospectuses downloaded/mailed, open day visits booked (in exchange for some sort of ‘discount’ to the potential student if they actually turn up to the open day), registrations/course applications made etc) advertising would play a major role in generating site revenue. If a single operator was running a suite of course choice sites, it would make sense for them to look at how cross-site exploitation of user data could be used to track users across sites and tune offerings for them. I suspect we’d also see the use of paid placement on some sites (putting results to the top of a search results listing based on payment rather than a more quality driven ranking algorithm), recreating some of the confusion of the early days of web searchengines.

I suspect there’d also be the opportunity for points-make-prizes competitions, and other giveaways…

Or like this maybe?

Ahem…

[Disclaimer: the opinions posted herein are, of course, barely even my own, let alone those of my employer.]

Written by Tony Hirst

May 29, 2012 at 12:38 pm

Posted in Stirring

Cognitive Waste and the Project Funding Bind

As I tweeted earlier today: “A problem with project funding is that you’re expected to know what you’re going to do in advance – rather than discover what can be done..”

This was prompted by reading a JISC ITT (Invitation to Tender) around coursedata: Making the most of course information – xcri-cap feed use demonstrators. Here’s an excerpt from the final call:

JISC is seeking to fund around 6-10 small, rapid innovation projects to create innovative, engaging examples that demonstrate the use of the #coursedata xcri-cap feeds (either directly, or via the JISC Aggregator API). These innovative examples will be shared openly through the JISC web site and events to promote the good practice that has been adopted.
13. The demonstrators could use additional data sources such as geolocation data to provide a mash-up, or may focus on using a single institutional feed to meet a specific need.
14. The demonstrators should include a clear and compelling use case and usage scenario.
15. The range of demonstrators commissioned will cover a number of different approaches and is likely to include examples of:
• an online prospectus, such as a specialist courses directory;
• a mobile app, such as a course finder for a specific geographical area;
• a VLE block or module, such as a moodle block that identifies additional learning opportunities offered by the host institution;
• an information dashboard, such as a course statistics dashboard for managers providing an analysis of the courses your institution offers mashed up with search trends from the institutional website;
• a lightweight service or interface, such as an online study group that finds peers based on course description;
• a widget for a common platform, such as a Google Gadget that identifies online courses, and pushes updates to the users iGoogle page.
16. All demonstrators should be working code and must be available under an open source licence or reusable with full documentation. Project deliverables can build on proprietary components but wherever possible the final deliverables should be open source. If possible, a community-based approach to working with open source code should be taken rather than just making the final deliverables available under an open source licence.
17. The demonstrators should be rapidly developed and be ready to use within 4 months. It is expected most projects would not require more than 30 – 40 chargeable person days.

In addition:

23. Funding will not be allocated to allow a simple continuation of an existing project or activity. The end deliverable must address a specific need that is accepted by the community for which it is intended and produce deliverables within the duration of the project funding.
24. There should be no expectation that future funding will be available to these projects. The grants allocated under this call are allocated on a finite basis. Ideally, the end deliverables should be sustainable in their own right as a result of providing a useful solution into a community of practice.

The call appears to be open to all comers (for example, sole traders) and represents a way of spending money on bootstrapping innovation around course data feeds using HEFCE funding, in a similar way to how the Technology Strategy money disburses money (more understandably?) to commercial enterprises, SMEs, and so on. (Although JISC isn’t a legal entity – yet – maybe we’ll start to see JISC trying to find ways in which it can start to act as a vehicle that generates returns from which it can benefit financially, eg as a venture funder, or as a generator of demonstrable financial growth?)

As with many JISC calls, the intention is that something “sustainable” will result:

22. Without formal service level agreements, dependency on third party systems can limit the shelf life of deliverables. For these types of projects, long term sustainability although always desirable, is not an expected outcome. However making the project deliverables available for at least one year after the end of the project is essential so opportunities are realised and lessons can be learned.

24. There should be no expectation that future funding will be available to these projects. The grants allocated under this call are allocated on a finite basis. Ideally, the end deliverables should be sustainable in their own right as a result of providing a useful solution into a community of practice.

All well and good. Having spent a shedload (technical term ;-) on getting institutions to open up their course data, the funders now need some uptake. (That there aren’t more apps around course data to date is partly my fault. The TSO Open Up Competition prize I won secured a certain amount of TSO resource to build something around course scaffolding code scaffolding data as held by UCAS (my proposal was more to do with seeing this data opened up as enabling data, rather than actually pitching a specific application…). As it turned out, UCAS (a charity operated by the HEIs, I think) were (still are?) too precious over the data to release it as open data for unspecified uses, so the prize went nowhere… Instead, HEFCE spent millions through JISC to get universities to open up course data (albeit probably more comprehensive than the UCAS data) instead…and now there’s an unspecified amount for startups and businesses to build services around the XCRI data. (Note to self: are UCAS using XCRI as an import format or not? If not, is HEFCE/JISC also paying the HEIs to maintain/develop systems that publish XCRI data as well as systems that publish data in an alternative way to UCAS?)

I think TSO actually did some work aggregating datasets around a, erm, model of the UCAS course data; so if they want a return on that work, they could probably pitch an idea for something they’ve already prepped and try to gt HEFCE to pay for it, 9 months on from when I was talking to them at their expense…

Which brings me in part back to my tweet earlier today (“A problem with project funding is that you’re expected to know what you’re going to do in advance – rather than discover what can be done..”), as well as the mantra I was taught way back when I was a research student, that the route to successful research bids was to bid to do work you had already done (in part because then you could propose to deliver what you knew you could already deliver, or could clearly see how to deliver…)

This is fine if you know what you’re pitching to do (essentially, doing something you know how to do), as opposed to setting out to discover what sorts of things might be possible if you set about playing with them. Funders don’t like the play of course, because it smacks of frivolity and undirectedness, even though it may be a deeply focussed and highly goal directed activity, albeit one where the goal emerges during the course of the activity rather than being specified in advance.

As it is, funders tend to fund projects. They tell bidders what they want, bidders tell funders back how they’ll do it (either something they’ve already done = guaranteed deliverable, paid for post hoc), or something they *think* they intend to do (couched in project management and risk assessment speak to mask the fact they don’t really know what’ll happen when they try to execute the plan, but that doesn’t really matter, because at the end of the day they have a plan and a set of deliverables against which they can measure (lack of) progress.) In the play world, you generally do or deliver something because that’s the point – you are deeply engaged in and highly focussed on whatever it is that you’re doing (you are typically intrinsically motivated and maybe also extrinsically motivated by whatever constraints or goals you have adopted as defining the play context/play world. During play, you work hard to play well. And then there’s the project world. In the project world, you deliver or you don’t. So what.

Projects also have overheads associated with them. From preparing, issuing, marking, awarding, tracking and reporting on proposals and funded projects on the fundrs’ side, to preparing, submitting, and managing the project on the other (aside from actually doing the project work – or at last, writing up what has previously been done in an appropriate way;-).

And then there’s the waste.

Clay Shirky popularised the notion of cognitive surplus to characterise creative (and often collaborative creative) acts done in folks’ free time. Things like Wikipedia. I’d characterise this use of cognitive surplus capacity as a form of play – in part because it’s intrinsically motivated, but also because it is typically based around creative acts.

But what about cognitive waste, such as arises from time spent putting together project proposals that are unfunded and then thrown away (why aren’t these bids, along with the successful ones, made open as a matter of course, particularly when the application is for public money from an applicant funded by public money?). (Or the cognitive waste associated with maintaining a regular blog… erm… oops…)

I’ve seen bids containing literature reviews that rival anything in the (for fee, paywall protected, subscription required, author/institution copyright waivered) academic press, as well as proposals that could be taken up, maybe in partnership, by SMEs for useful purpose, rather than academic partners for conference papers), to time spent pursuing project processes, milestones and deliverables for the sole reason that they are in the plan that was defined before the space the project was pitched in to is properly through engaging with it, rather than because they continue to make sense (if indeed they ever did). (And yes, I know that the unenlightened project manager who sees more merit in trying to stick to the project plan and original deliverables, rather than pivoting if a far more productive, valuable or useful opportunity reveals itself, is a mythical beast…;-).

Maybe the waste is important. Evolution is by definition wasteful process, and maybe the route to quality is through a similar sort of process. Maybe the time, thought and effort that goes into unsuccessful bids really is cognitive waste, bad ideas that don’t deserve to be shared (and more than that, shouldn’t be shared because they are dangerously wrong). But then, I’m not sure how that fits in with project funding schemes that are over-subscribed and even highly rated proposals (that would ordinarily receive funding) are rejected, whereas in an undersubscribed call (maybe because it is mis-positioned or even irrelevant), weak bids (that ordinarily wouldn’t be considered) get funding.

Or maybe cognitive waste arises from a broken system and broken processes, and really is something valuable that is being wasted in the sense of squandered?

Right – rant over, (no) (late)lunchtime over… back to the “work” thing, I guess…

PS via @raycorrigan: “Newton, Galileo, Maxwell, Faraday, Einstein, Bohr, to name but a few; evidence of paradigm shifting power of ‘cognitive waste’” – which is another sense of “waste” I hadn’t considered, which is waste (as in loss, or loss to an organisation) of good ideas through rejecting or not supporting the development of a particular proposal or idea..?

Written by Tony Hirst

May 25, 2012 at 2:16 pm

Posted in Stirring

Tagged with

Mapping the Tesco Corporate Organisational Sprawl – An Initial Sketch

A quick sketch, prompted by Tesco Graph Hunting on OpenCorporates of how some of Tesco’s various corporate holdings are related based on director appointments and terminations:

The recipe is as follows:

- grab a list of companies that may be associated with “Tesco” by querying the OpenCorporates reconciliation API for tesco
- grab the filings for each of those companies
- trawl through the filings looking for director appointments or terminations
- store a row for each directorial appointment or termination including the company name and the director.

You can find the scraper here: Tesco Sprawl Grapher

import scraperwiki, simplejson,urllib

import networkx as nx

#Keep the API key [private - via http://blog.scraperwiki.com/2011/10/19/tweeting-the-drilling/
import os, cgi
try:
    qsenv = dict(cgi.parse_qsl(os.getenv("QUERY_STRING")))
    ockey=qsenv["OCKEY"]
except:
    ockey=''

rurl='http://opencorporates.com/reconcile/gb?query=tesco'
#note - the opencorporates api also offers a search:  companies/search
entities=simplejson.load(urllib.urlopen(rurl))

def getOCcompanyData(ocid):
    ocurl='http://api.opencorporates.com'+ocid+'/data'+'?api_token='+ockey
    ocdata=simplejson.load(urllib.urlopen(ocurl))
    return ocdata

#need to find a way of playing nice with the api, and not keep retrawling

def getOCfilingData(ocid):
    ocurl='http://api.opencorporates.com'+ocid+'/filings'+'?per_page=100&api_token='+ockey
    tmpdata=simplejson.load(urllib.urlopen(ocurl))
    ocdata=tmpdata['filings']
    print 'filings',ocid
    #print 'filings',ocid,ocdata
    #print 'filings 2',tmpdata
    while tmpdata['page']<tmpdata['total_pages']:
        page=str(tmpdata['page']+1)
        print '...another page',page,str(tmpdata["total_pages"]),str(tmpdata['page'])
        ocurl='http://api.opencorporates.com'+ocid+'/filings'+'?page='+page+'&per_page=100&api_token='+ockey
        tmpdata=simplejson.load(urllib.urlopen(ocurl))
        ocdata=ocdata+tmpdata['filings']
    return ocdata

def recordDirectorChange(ocname,ocid,ffiling,director):
    ddata={}
    ddata['ocname']=ocname
    ddata['ocid']=ocid
    ddata['fdesc']=ffiling["description"]
    ddata['fdirector']=director
    ddata['fdate']=ffiling["date"]
    ddata['fid']=ffiling["id"]
    ddata['ftyp']=ffiling["filing_type"]
    ddata['fcode']=ffiling["filing_code"]
    print 'ddata',ddata
    scraperwiki.sqlite.save(unique_keys=['fid'], table_name='directors', data=ddata)

def logDirectors(ocname,ocid,filings):
    print 'director filings',filings
    for filing in filings:
        if filing["filing"]["filing_type"]=="Appointment of director" or filing["filing"]["filing_code"]=="AP01":
            desc=filing["filing"]["description"]
            director=desc.replace('DIRECTOR APPOINTED ','')
            recordDirectorChange(ocname,ocid,filing['filing'],director)
        elif filing["filing"]["filing_type"]=="Termination of appointment of director" or filing["filing"]["filing_code"]=="TM01":
            desc=filing["filing"]["description"]
            director=desc.replace('APPOINTMENT TERMINATED, DIRECTOR ','')
            director=director.replace('APPOINTMENT TERMINATED, ','')
            recordDirectorChange(ocname,ocid,filing['filing'],director)

for entity in entities['result']:
    ocid=entity['id']
    ocname=entity['name']
    filings=getOCfilingData(ocid)
    logDirectors(ocname,ocid,filings)

The next step is to graph the result. I used a Scraperwiki view (Tesco sprawl demo graph) to generate a bipartite network connecting directors (either appointed or terminated) with companies and then published the result as a GEXF file that can be loaded directly into Gephi.

import scraperwiki
import urllib
import networkx as nx

import networkx.readwrite.gexf as gf

from xml.etree.cElementTree import tostring

scraperwiki.sqlite.attach( 'tesco_sprawl_grapher')
q = '* FROM "directors"'
data = scraperwiki.sqlite.select(q)

DG=nx.DiGraph()

directors=[]
companies=[]
for row in data:
    if row['fdirector'] not in directors:
        directors.append(row['fdirector'])
        DG.add_node(directors.index(row['fdirector']),label=row['fdirector'],name=row['fdirector'])
    if row['ocname'] not in companies:
        companies.append(row['ocname'])
        DG.add_node(row['ocid'],label=row['ocname'],name=row['ocname'])   
    DG.add_edge(directors.index(row['fdirector']),row['ocid'])

scraperwiki.utils.httpresponseheader("Content-Type", "text/xml")


writer=gf.GEXFWriter(encoding='utf-8',prettyprint=True,version='1.1draft')
writer.add_graph(DG)

print tostring(writer.xml)

Saving the output of the view as a gexf file means it can be loaded directly in to Gephi. (It would be handy if Gephi could load files in from a URL, methinks?) A version of the graph, laid out using a force directed layout, with nodes coloured according to modularity grouping, suggests some clustering of the companies. Note the parts of the whole graph are disconnected.

In the fragment below, we see Tesco Property Nominees are only losley linked to each other, and from the previous graphic, we see that Tesco Underwriting doesn’t share any recent director moves with any other companies that I trawled. (That said, the scraper did hit the OpenCorporates API limiter, so there may well be missing edges/data…)

And what is it with accountants naming companies after colours?! (It reminds me of sys admins naming servers after distilleries and Lord of the Rings characters!) Is there any sense in there, or is arbitrary?

Written by Tony Hirst

April 12, 2012 at 3:56 pm

Tesco Graph Hunting on OpenCorporates

A quick lunchtime post on some thoughts around constructing corporate graphs around OpenCorporates data. To ground it, consider a search for “tesco” run on gb registered companies via the OpenCorporates reconciliation API.

{"result":[{"id":"/companies/gb/00445790", "name":"TESCO PLC", "type":[{"id":"/organization/organization","name":"Organization"}], "score":78.0, "match":false, "uri":"http://opencorporates.com/companies/gb/00445790"}, {"id":"/companies/gb/05888959", "name":"TESCO AQUA (FINCO1) LIMITED", "type":[{"id":"/organization/organization", "name":"Organization"}], "score":71.0, "match":false, "uri":"http://opencorporates.com/companies/gb/05888959"}, { ...

Some or all of these companies may or may not be part of the same corporate group. (That is, there may be companies in that list with Tesco in the name that are not part of the group of companies associated with a major UK supermarket.)

If we treat the companies returned in that list as one class of nodes in a graph, we can start to construct a range of graphs that demonstrate linkage between companies based on a variety of factors. For example, a matching address for a registered, post off box mediated, address in an offshore tax haven suggests there may be a weak tie at least between companies:

(Alternatively, we might construct bipartite graphs containing company nodes and address nodes, for example, then collapse the graph about common addresses.)

Shared directors would be another source of linkage, although at the moment, I don’t think OpenCorporates publishes directors associated with UK companies (I suspect that data is still commercially licensed?). However, there is associated information available in the OpenCorporates database already…. For example, if we look at the various company filings, we can pick up records relating to director appointments and terminations?

By monitoring filings, we can then start to build up a record of directorial involvement with companies? From looking at the filings, it also suggests that it would make sense to record commencement and cessation dates for directorial appointments…

There may also be weak secondary evidence linking companies. For example, two companies that file trademarks using the same agent have a weak tie through that agent. (Of course, that agent may be acting for two completely independent companies.)

If we weight edges between nodes according to the perceived strength of a tie and then lay out the graph in a way that is sensitive to the number of weight of edge connections between company nodes, we may be able to start mapping out the corporate structure of these large, distributed corporations, either in network map terms, or maybe by mapping geolocated nodes based on registered addresses; and then we can start asking questions about why these distributed corporate entities are structured the way they are…

PS note to self – OpenCorporates API limit with key: 1000/hr, 10k/day

Written by Tony Hirst

April 12, 2012 at 12:36 pm

Posted in Anything you want, Stirring

Tagged with

Autodiscoverable Feeds and UK HEIs (Again…)

It’s that time of year again when Brian’s banging on about IWMW, the Instituional[ised?] Web Managers’ Workshop, and hence that time of year again when he reminds me* about my UK HE Feed Autodiscovery app that trawls through various UK HEI home pages (the ones on .ac.uk, rather than the one you get by searching for a uni name in Google;-)

* that is, tells me the script is broken and, by implication, gently suggests that I should fix it…;-)

As ever, most universities don’t seem to be supporting autodiscoverable feeds (neither are many councils…), so here are a few thoughts about what feeds you might link to, and why…

- news feeds: the canonical example. News feeds can be used to pipe news around various university websites, and also syndicate content to any local press or hyperlocal news sites. If every UK HEI published a news feed that was autodiscoverable as such, it would be trivial to set up a UK universities aggregated newswire.

- research announcements: I was told that one reason for putting out press releases was simply to build up an institutional memory/archive of notable events. Many universities run research newsletters that remark on awarded grants. How about a “funded research” feed from each university detailing grant awards and other research funding. Again, at a national level, this could be aggregated to provide a research funding newswire, as well as contribtuing data to local archives of research funding success.

- jobs: if every UK HEI published a jobs/vacancies RSS feed, it would trivial to build an aggregator and let people roll their own versions of jobs.ac.uk.

- events: universities contribute a lot to local culture through public talks and exhibitions. Make it easy for the local press and hyperlocal news sites to syndicate this info, and add events to their own aggregated “what’s on” calendars. (And as well as RSS, give ‘em an iCal feed for your events.)

- recent submissions to local repository: provide a feed listing recent submissions to the local research output/paper repository (and/or maybe a feed of the most popular downloads); if local feeds are you thing, the library quite possibly makes things like recent acquisition feeds available…

- YouTube uploads: you might was well add an autodiscoverable feed to your university’s recent uploads on YouTube. If nothing else, it contributes an informal ownership link to the web for folk who care about things like that.

- your university Twitter feed: if you’ve got one. I noticed Glasgow Caledonian linked to their Twitter feed through an autodiscoverable link on their university homepage.

- tenders: there’s a whole load of work going on in gov at the moment regarding transparency as it relates to procurement and tendering. So why not get open with your procurement and tendering data, and increase the chances of SMEs finding out what you’re tendering around. If the applications have to go through a particular process, no problem: link to the appropriate landing page in each feed item.

- energy data: releasing this data may well become a requirement in the not so far off future, so why not get ahead of the game, e.g. as Lincoln are starting to do (Lincoln U energy data)? If everyone was publishing energy data feeds, I’m sure DevCSI hackday folk would quickly roll together something like the aggregating service built by college student @issyl0 out of a Rewired State hack that pulls together UK gov department energy data: GovSpark

- XCRI-CAP course marketing data feeds: JISC is giving away shed loads of cash to support this, so pull your finger out and get the thing published.

- location data: got a KML feed yet? If not, why not? e.g. Innovations in Campus Mapping

PS the backend of my RSS feed autodiscovery app (founded: 2008) is a Yahoo pipe. Just because, I thought I’d take half an hour out to try and build something related on Scraperwiki. The code is here: UK University Autodiscoverable RSS feeds. Please feel free to improve or, fork it, etc. University homepage URLs are identified by scraping a page on the Universities UK website, but I probably should use a feed from the JISC Monitoring Unit (e.g. getting UK University location/contact data).

PPS this could be handy for some folk – the code that runs the talks@cam events site: http://source.caret.cam.ac.uk/svn/projects/talks.cam/. (Thanks Laura:-) – does it do feeds nicely now?! Related: Keeping Up With Events, a quickly hacked app from my Arcadia project that (used to) aggregate Cambridge events feeds.)

Written by Tony Hirst

July 26, 2011 at 6:59 pm

Getting Access to University Course Code Data (or not… (yet…))

A couple of weeks or so ago, having picked up the TSO OpenUp competition prize for suggesting that it would be a Good Thing for UCAS/university course code data to be made available, I had a meeting with the TSO folk to chat over “what next?” The meeting was an upbeat one with a plan to get started as soon as possible with a scrape of the the UCAS website… so what’s happened since…?

First up – a reading of the UCAS website Terms and Conditions suggests that scraping is a no-no…

6. Intellectual property rights
e. Copying, distributing or any use of the material contained on the website for any commercial purpose is prohibited.
f. You may not create a database by systematically downloading substantial parts of the website

(In the finest traditions of the web, you aren’t allowed to deep link into the site without permission either: 6.c inks to the website are not permitted, other than links to the homepage for your personal use, except with our prior written permission. Links to the website from within a frameset definition are not permitted except with our prior written permission.)

So, err, I guess my link to the terms and conditions breaks those terms and conditions? Oops…;-) Should I be sending them something like this do you think?

Dear enquiries@ucas.ac.uk,
As per your terms and conditions, (paragraph 6 c) please may I publish a link to your terms and conditions web page [ http://www.ucas.com/terms_and_conditions ] in a blog post I am writing that, in part, refers to your terms and conditions?
Luv'n'hugs,
tony

As a fallback, I put a couple of trial balloon FOI requests in to a couple of universities asking for the course names and UCAS course codes for courses offered in 2010/11, along with the search keywords associated with each course (doh! I did it again, deep linking into the UCAS site…)

PS Please may I also link to the page describing course search keywords [ http://www.ucas.com/he_staff/courses/coursesearchkeywords ] ?

The first request went to the University of Southampton, in part because I knew that they already publish chunks of the data (as data) as part of their #opensoton Open Data initiative. (This probably means I was abusing the FOI system, but a point maybe needed to be made…?!;-) The second request was put in to the University of Bristol.

The requests were of the form:

I would be grateful if you could send me in spreadsheet, machine readable electronic form or plain text a copy of the course codes, course titles and search keywords for each course as submitted to UCAS for the 2010-2011 (October 2010) student entry.

If possible, would you also provide HESA subject category codes associated with each course.

So how did I get on?

Bristol’s response was as follows:

On discussion with our Admissions and Student Information teams, it appears that the University does not actually hold this data – it is held on a UCAS database. UCAS are not currently subject to the Freedom of Information Act (they will be in due course) but it may be worth talking to them directly to see if they are willing to assist.

And Southampton’s FOI response?

Course codes and titles may be found here: http://www.soton.ac.uk/corporateservices/foi/request-66210-6124d691.pdf Keywords were not held by the University – you should inquire with UCAS (http://www.ucas.com). HESA subject category codes may be found here: http://www.hesa.ac.uk/index.php/content/view/1806/296/

So what did I learn?

  1. I don’t seem to have made it clear enough to Southampton that I wanted the the 2-tuple (course code, HESA code) for each course. So how should I have asked for that data (the response pointed me to the list of all HESA codes. What I wanted was, for each course code, the course code/HESA code pair).
  2. Generalising from an example of one;-), there seems to be a disconnect between FOI and open data branches of organisations. In my ideal world, the FOI person (an advocate for the person making the request) would also be on good terms with the Open Data team in the organisation, if not a data wrangler themselves. For data requests, the FOI person would make sure the data is released as open data as part of the process of fulfilling the request and then refer the person making the request to the open data site (see also: Open Data Processes – Taps, Query Paths/Audit Trails and Round Tripping). Southampton have part of this process already – the course data is in a PDF on the their site and I was referred to it. (Note that the PDF is not just any PDF – have a look at it! – rather than the spreadsheet, machine readable electronic form or plain text I requested, even though @cgutteridge had posted a link to the SPARQL opendata query for the course code/UCAS code information I’d requested as a reply to my FOI request on the WhatDoTheyKnow site.)
  3. Universities don’t necessarily have any record of the search keywords they associate with the courses they post on UCAS. The UCAS website suggests that (doh!) “[r]ecent analysis of unique IP address use of the UCAS Course Search indicates that the subject search is by far the most popular of the 3 search options currently available”, such that “[w]hen an applicant uses our Course Search facility to search for available courses, they can choose a keyword by which to search, known as the ‘subject search’.” Which is to say, universities have no local record of the terms they use to describe courses that are the the primary way of discovering their courses on UCAS? Blimey… (I wonder how much universities spend on Google AdWords for advertising particular courses on their own course prospectus websites and how they go about selecting those terms?)
  4. Asking for a machine readable “data as data” response has no teeth at the current time. I don’t know if the Protection of Freedoms bill clause that “extends Freedom of Information rights by requiring datasets to be available in a re-usable format” will change this? It seems like it might?

    Where—
    (a) an applicant makes a request for information to a public authority in respect of information that is, or forms part of, a dataset held by the public authority, and
    (b) on making the request for information, the applicant expresses a preference for communication by means of the provision to the applicant of a copy of the information in electronic form, the public authority must, so far as reasonably practicable, provide the information to the applicant in an electronic form which is capable of re-use.

  5. So what next? UCAS is a charity that appears to be operated by, for, and on behalf of UK Higher Education (e.g. UCAS Directors’ Report and Accounts 2009). Whilst not FOIable yet, it looked set to become FOIable from October 2011 (Ministry of Justice: Greater transparency in Freedom of Information), though I haven’t been able to find the SI and commencement date that enact this…?). IF it does become FOIable, we may be able to get the data out that way (although memories of the battle between open data advocates and the Ordnance Survey come to mind…) Hopefully, though, we’ll be able to get the data open by more amicable means before then…:-)

    PS a couple of other things that I’ve been dipping into relating to this project. Firstly, the UCAS Business Plan 2009-2012 (doh!):

    PPS Please may I also link to your Corporate Business Plan 2009-2012 [ http://www.ucas.com/documents/corporate/corpbusplan09-12.pdf ]

    Secondly, the Cabinet Office’s “Better Choices: Better Deals” strategy document [PDF], which as well as its “MyData” right to personal data initiative, also encourages business to put their information (and data…) to work. Whether or not you agree that more information may help to make for better choices from potential students, or that comparison sites have a role to play in this, the UK government appears to believe it and looks set to support the development of businesses operating in this area. For example:

    Effective consumer choices are also important in the public sector – such as decisions about what and where to study.
    However, unlike in private markets, public services are generally:
    ● Free at the point of delivery, so prices do not give us clues about quality or popularity.
    ● Not motivated by profits, so there is little incentive to highlight differences and encourage switching.
    ● Supplied under a universal service obligation, such that they serve a particularly broad range of users, from the very informed to the highly vulnerable.
    In the same way that comparison and feedback sites have developed for private markets, some choice-tools have already emerged for public services. For example, parents and prospective students can use league tables to compare school and university performance, while patients can access websites comparing waiting times for treatments across different healthcare providers, and feedback from fellow consumers about the performance of a local GP practice. Their role is likely to become more important in future as public service markets are opened up and there is scope for further choice-tools to be developed [Better Choices: Better Deals, p. 32]

    If you’re looking to put a bid or business plan together based on using public data as a basis for comparison services, the Better Choices document has more than a few quotable sections;-)

    [Related: Course Detective metasearch/custom search across UK University prospectus websites]

Written by Tony Hirst

April 26, 2011 at 12:58 pm

Posted in Data, Stirring, Thinkses

Tagged with , , , ,

Follow

Get every new post delivered to your Inbox.

Join 729 other followers