www.gov.uk – Replacing the master (dot com) of Previous Government Departmental Web Domains?

By the by, I ran a search on the deprecated dwp.gov.uk website earlier today (Google took me there originally, I think, rather than to the new site reached via http://www.gov.uk/dwp):

search on dwp.gov.uk

and ended up on a search results page with the URL http://dwp.gov.uk.master.com/texis/master/search/?q=sharing+data+local+authority&s=SS:

dwp search

(The results appear to be broken – on the first link at least, the redirect from the results page goes to a largely irrelevant link on the new gov.uk site.)

Hmm…, master.com?

master.com homepage

Ooh – slick… ‘ere, gov, wanna buy a new motorwebservice?

DRIP… Letter to My MP

Dear Andrew Turner,

I am writing to you today to express concern about the way in which the Data Retention and Investigatory Powers Bill [1] is being rushed through Parliament without appropriate time allowed for scrutiny of the Bill or members to elicit expert and informed opinion about what the consequences of the Bill might be.

I do not know if you have read the Bill [1], the supporting documentation (explanatory notes and impact assessment) [2], or the ECJ ruling [3] that prompted the emergency legislation, or how you intend to vote in the matter of the Bill. If you have not read the supporting documents, but you do intend to vote in support of the Bill, I would like to ask by what rationale you came to that decision? Because it would not be in my name as a constituent and you would not have my support in the matter.

If you do not have time to read through the official documents, may I ask what other sources of information you turned to in forming your opinion. For your information, [4] considers some of the issues raised in the impact assessment document.

Informed opinion appears to differ in several important respects in the extent to which it believes the Bill may introduce new or extended powers or extend the scope of regulation, rather than just securing previous legislation, from the public statement that introduced the bill, and the Home Secretary’s comments in the HoC Home Affairs Committee today. This suggests that more time for considering this Bill is required – if the House is not clear about what the legislation says, it should not pass it. If it is clear, how does it counter the claims raised by the informed dissenters?

As I have written to you previously, I would also like to express concern once again about the cavalier and ill-informed way in which the Government introduce and attempt to pass legislation particularly in respect of “digital” matters.

Yours sincerely,

Tony Hirst


Open Data For Transparency – Contract Compliance?

Doing a tab sweep just now, I came across a post describing a Policy note issued on contracts and compliance with transparency principles by the Crown Commercial Service [Procurement Policy Note – Increasing the Transparency of Contract Information to the Public – Action Note 13/15 31 July 2015 ] that requires “all central government departments including their Executive Agencies and Non-Departmental Public Bodies (NDPBs)” (and recommends that other public bodies follow suit) to publish a “procurement policy note” (PPN) “in advance of a contract award, [describing] the types of information to be disclosed to the public, and then to publish that information in an accessible form” for contracts advertised after September 1st, 2015.

The publishers should “explore [make available, and update as required] a range of types of information for disclosure which might typically include:-

i. contract price and any incentivisation mechanisms
ii. performance metrics and management of them
iii. plans for management of underperformance and its financial impact
iv. governance arrangements including through supply chains where significant contract value rests with subcontractors
v. resource plans
vi. service improvement plans”

The information so released may perhaps provide a hook around which public spending (transparency) data could be used to cross-check contract delivery. For example, “[w]here financial incentives are being used to drive performance delivery, these may also be disclosed and published once milestones are reached to trigger payments. Therefore, the principles are designed to enable the public to see a current picture of contractual performance and delivery that reflects a recent point in the contract’s life.” Spotting particular payments in transparency spending data as milestone payments could perhaps be used to cross-check back that those milestones were met in an appropriate fashion. Or where dates are associated in a contract with particular payments, this could be used to flag-up when payments might be due to appear in spending data releases, and raise a red flag if they are not?

Finally, the note further describes how:

[i]n-scope organisations should also ensure that the data published is in line with Open Data Principles. This means ensuring the data is accessible (ideally via the internet) at no more than the cost of reproduction, without limitations based on user identity or intent, in a digital, machine-readable format for interoperation with other data and free of restriction on use or redistribution in its licensing conditions

In practical terms, of course, how useful (if at all) any of this might be will in part determined by exactly what information is released, how, and in what form.

It also strikes me that flagging up what is required of a contract when it goes out to tender may still differ from what the final contract actually states (and how is that information made public?) And when contracts are awarded, where’s the data (as data) that clearly states who it was awarded to, in a trackable form, etc etc. (Using company numbers in contract award statements, as well as spending data, and providing contract numbers in spending data, would both help to make that sort of public tracking easier…)

Tutum Cloud Becomes Docker Cloud

In several previous posts, I’ve shown how to launch docker containers in the cloud using Tutum Cloud, for example to run OpenRefine, RStudio, or Shiny apps in the cloud. Docker bought Tutum out several months ago, and now it seems that they’re running the tutum environment as Docker Cloud (docs, announcement).


Tutum – as was – looks like it’s probably on the way out…


The onboarding screen confirms that the familiar Tutum features are available in much the same form as before:


The announcement post suggests that you can get a free node and one free repository, but whilst my Docker hub images seemed to have appeared, I couldn’t see how to get a free node. (Maybe I was too quick linking my Digital Ocean account?)

As with Tutum, (because it is, was, tutum!), Docker Cloud provides you with the opportunity to spin up one or more servers on a linked cloud host and run containers on those servers, either individually or linked together as part of a “stack” (essentially, a Docker Compose container composition). You also get an image repository within Docker Cloud – mine looks as if it’s linked to my DockerHub repository:


A nice feature of this is that you can 1-click start a container from your image repository if you have a server node running.

The Docker Cloud service currently provides a rebranding of the Tutum offering, so it’ll be interesting to see if product features continue to be developed. One thing I keep thinking might be interesting is a personal MyBinder style service that simplifies the deploy to Tutum (as was) service and allows users to use linked hosts and persistent logins to 1-click launch container services with persistent state (for example, Launch Docker Container Compositions via Tutum and Stackfiles.io – But What About Container Stashing?). I guess this would mean linking in some cheap storage somehow, rather than having to keep server nodes up to persist container state? By the by, C. Titus Brown has some interesting reflections on MyBinder here: Is mybinder 95% of the way to next-gen computational science publishing, or only 90%?

If nothing else, signing up to Docker Cloud does give you a $20 Digital Ocean credit voucher that can also be applied to pre-exisiting Digital Ocean accounts:-) (If you haven’t already signed up for Digital Ocean but want to give it a spin, this affiliate link should also get you $10 free credit.)

PS as is the way of these things, I currently run docker containers in the cloud on my own tab (or credit vouchers) rather than institutional servers, because – you know – corporate IT. So I was interested to see that Docker have also recently launched a Docker DataCenter service (docs), and the associated promise of “containers-as-a-service” (CaaS), that makes it easy to offer cloud-based container deployment infrastructure. Just sayin’…;-)

PPS So when are we going to get Docker Cloud integration in Kitematic, or a Kitematic client for Docker Cloud?

Calling an OData Service From Python – UK Parliament Members Data Platform

Whilst having a quick play producing Slack bots and slash commands around the UK Parliament APIs, I noticed (again) that the Members data platform has an OData endpoint.

OData is a data protocol for querying online data services via HTTP requests although it never really seemed to have caught the popular imagination, possibly because Microsoft thought it up, possibly because it seems really fiddly to use…

I had a quick look around for Python client/handler for it, and the closest I came was the pyslet package. I’ve posted a notebook showing my investigations to date here: Handling the UK Parliament Members Data Platform OData Feed, but it seems really clunky and I’m not sure I’ve got it right! (There doesn’t seem to be a lot of tutorial support out there, either?)

Here’s an example of the sort of mess I got myself in:


To make the Parliament OData service more useful needs a higher level Python wrapper, I think, that abstracts a bit further and provides some function calls that make it a tad easier (and natural) to get at the data. Or maybe I need to step back, have a read of the OData blocks, properly get my head around the pyslet OData calls, and try again!

Simple Live Timing Data Scraper…

A couple of weeks ago, I noticed an F1 live timing site with an easy to hit endpoint… here’s the Mac commandline script I used to grab the timing info, once every five seconds or so…

mkdir f1_silverstone
i=1; sleep 900; while true ; do curl http://www.livesportstreaming24.com/live.php >> f1_silverstone/f1output_race_${i}.txt ;i=$((i+1)); sleep 5 ; done

Now I just need to think what I’m going to do with the data! Maybe an opportunity to revisit this thing and try out some realtime dashboard widget toys?

PS to get the timestamp of each file in python:

import os