In April 2010, Bloomberg Businessweek Magazine asked Are Sportswriters Really Necessary?, describing how Narrative Science, a company that incorporated at the start of that year and spun out off the back of the Stats Monkey project had teamed up with the Big Ten Network to produce automatically generated sports reports, a relationship that presumably continues to this day.

A year later, and Forbes magazine produced a report in June 2011 about GameChanger and Narrative Science: Fulfilling the Heretofore Unrealized Demand for Stilted Stories About Children’s Games, describing a tie-up between Narrative Science and GameChanger, a company that produces a scorekeeping app that allows sports fans, parents and coaches to capture data about a match.

(What other companies/apps are out there for crowdsourcing sports analytics in this way, I wonder?)

Using GameChanger data and narrative Science story generation tools, it was possible to automate the creation of match reports for small number audiences. I don’t know if these stories used to be freely accessible, but today the match reports appear to take the form of paywalled notion of recap stories.

Paywall aside, examples of other stories generated by Narrative Science using GameChanger data can be found using a simple web search on the phrase “Powered by Narrative Science and GameChanger Media”

You can also just search for the byline, as for example it appears in this report:

In passing, it’ll be interesting to see how automatically generated stories start to feed into the glitch aesthetic (h/t @danmcquillan for introducing me to this phrase and the related notion of the new aesthetic in his presentation at #opentech last week).

September 2011 saw a media outlook report from Mediabistro’s Media Jobs Daily noting that Narrative Science’s ‘Robot Journalists’ Now Tackling Real Estate. The story links through to a page on Builder Online that provides a summary report of housing data for various US cities.

What this example, and the GameChanger example, show is how the generation of timely text stories can be automated on top of the regularly updated datasets. The use of natural language interpretive text to describe patterns observed in the underlying data presumably also has SEO benefits.

That same month, September 2011, saw another stats-to-insight company, again emerging from the automated interpretation of sports data, renaming itself from StatSheet to Automated Insights. Today, Statsheet continues to publish game recaps combining short natural language summaries with statistical charts, all of which are presumably automatically generated. Within a year, the parent company, Automated Insights had scaled up and begun publishing recaps for Yahoo!’s fantasy sports matches.

More recently, Automated Insights have started producing realtime content feeds to support sports commentators – Real-time Insights for MLB – as well as feeding consumers via the stat.us powered Twitter feeds.

(See also: yseop, a French company that generates automated reports from data. [Any more?])

Fast forward to the start of 2013, and Narrative Science started publishing human readable prose reports based on US schools data (ProPublica: How To Edit 52,000 Stories at Once). They’re also doing a lot more work with financial reporting, for example with Forbes as well as for financial services clients, as this interview with Narrative Science’s Stuart Frankel describes.

Generating human readable reports from Google Analytics data and dashboards also appears to be a hot topic, with both Narrative Science (Automated Insight From Google Analytics With Quill) and Automated Insights (With Site Ai, Automated Insights Provides A Cliffs Notes Version Of Your Web Analytics) recently developing tools around this topic.

What I thought was particularly interesting about the ProPublica example was how it suggests a possible widespread future use of “automatically generated insight” pulling out headline interpretations from open data sets, as touched on in this great introductory technical presentation by Narrative Science’s Larry Adams (which also happens to mention the possibility of Narrative Science offering platform services via an API…? It also mentions work with the NHS?):

At one point during that presentation, Larry Adams suggests that Narrative Science use small set of narrative templates or story types (“the horserace” for example, or “top 10″) to frame the construction of their stories, as well as mentioning the sorts of feature that they look for within a data set (trends and changes in trends, for example, or outliers). Another presentation, this time by Narrative Science’s Kris Hammond also hints at some of the features they look for in data: “inflexion points, trends, correlations”.

So what sorts of techniques might we use ourselves to start generating the insights that we might be able to work up into simple narrative sentences, at least for starters?

Top 10, bottom 5 are easy pickings if we can rank the data somehow. I thought this trick for detecting inflexions by coding a time series symbolically and then using a regular expression to detect features was really interesting: Finding patterns in time series using regular expressions. And I wonder, how does the OpenSecrets anomaly tracker define the anomalies it detects?

Other posts you might be interested in:
- The Tesco Data Business – Notes on “Scoring Points”
- More Remarks on the Tesco Data Play

- killing Google Reader does not kill RSS/there was no “malicious intent” mapping out the Reader/RSS strategy:

A nice phrase in an #opentech talk yesterday was that we (technologists and engineers and data scientists, for example) have to “act responsibly”. Google Reader helped popularise feed reading when some of us were hopeful for its future (“We ignore RSS at OUr Peril”), and as such attracted many readers away from other clients (myself included), with the result that competition was harder (“compete against Google? Hmm… maybe not…”). Google Reader’s infrastructure and unofficial APIs enabled folk to build services off the back of the Google Reader infrastructure turning it into de facto infrastructure for other peoples’ applications and services. (Remember: the Google Maps API was unofficial at first). There aren’t many OPML bundlers out there, for example, but for hackers into appropriating tech Google Reader is one. Since I moved away from Google Reader (to theoldreader) I haven’t used Flipboard so much, which as far as I was concerned was using Reader essentially as infrastructure. Caveat emptor, I guess, for developers building on top of other companies services (as many Twitter and Facebook app developers keep discovering).

With Feedburner, Google bought up a service that acted as a proxy, taking public syndication feeds, instrumenting them with analytics, and then encouraging the people taking up the syndicated content to subscribe to the Feedburner feed. Where RSS and Atom were designed to support syndication between independent parties, Feedburner – and then Google – insinuated itself between those parties. By replacing self-controlled feeds as the subscription endpoint with Google controlled endpoints, publishers gave up control of their syndication infrastructure. With Google losing interest in open syndication feeds as it pursues its own closed content network agenda, we are faced with a situation whereby Google can potentially trash a widespread syndication infrastructure that would have remained resilient if Google hadn’t insinuated itself into it. Or if we hadn’t been so stupid as to simplistically accept it’s overtures.

Hmmm… thinks… do we need a Google users’ motto? Don’t be stupid perhaps…?!

I applaud Google for developing the services it does, getting them to scale and opening up API access. But as these services become de facto infrastructure, the question of how Google acknowledges any responsibility, that flows from this (even if this responsibility is incorrectly assumed) becomes an issue. Responsibilities arise in other areas too, of course. Such as taxation and corporate transparency. But that’s another issue. (Would Google act differently if its motto was “Be responsible” or “Act responsibly” rather than “Don’t be evil”? It strikes me that “Act responsibly” could work as a motto for both companies and their users?)

It seems to me that with Google+, Google is not adopting open syndication standards in two ways: not using it “internally”, and not making feeds publicly available. There may be good technical reasons for the first, but by the second Google is *not allowing* its community members to participate in a open content syndication network/system. Google’s choice, but I’m not playing.

Google is not killing the open standards by closing off access to them in commercial licensing terms, but it may contribute to stifling their adoption by adopting alternative standards that others feel they have to adopt because of the influence Google has on web traffic.

Consider this other way of looking at it – Google is presumably trying to get other parties to adopt WebP by developing it as an openstandard. Google assumes that it can drive adoption of this as a web standard by adopting it itself. In terms of argumentation, it doesn’t follow that by not adopting something Google can prevent it being adopted, (i.e. not adopting or by stopping its own use of a standard, Google kills it generally) but people follow bad logic all the time (and if they follow Google for their technology choices, or have a technology model based on being parasitic on Google infrastructure, Google’s dropping of a standard effectively kills it for those people) …

- control of what we see

Google makes money by putting ad-links in front of eyeballs that people click on. By presenting “relevant” ads, Google presumably tries to maximise the click-thru rate so that it can make more money per displayed link.

To encourage you to spend your attention on pages that Google controls, Google has adopted the idea that by presenting you (and me; us) with “relevant” content, we are likely to remain engaged. With Google web search, the relevance of search results supposedly attracts us back to the Google search tool. With services such as Google now, Google pre-emptively tries to present you with information it thinks you need, presumably based on predictive models of sequences of action that other people (or you yourself) have demonstrated in the past.

I’m not really up on behavioural psychology models, but I have a vague memory that intermittent reinforcement schedules were demonstrated to be one of the more effect modes of behaviourist training/operant conditioning. So I wonder: how effective are predictive intermittent positive reinforcement schedules. (You get the idea, right? We’re pigeons that peck at Android phones and Google is the experimenter trying to get us to peck the right way, by reinforcing us every now and again by satisfying out intent. That is, has there been in a flip away from Google using us to provide reinforcement training signals to its algorithms in to a situation in which we have become Google’s experimental lab rats that are coupled in a series of ongoing experiments that train us and its algorithms, jointly, together, to maximise… something…)

There is a danger, I think, in Google chasing the “relevance” thing too far, seeing the maximisation of whatever conversion metrics it decides on as being a sign that it has “got things right” for us, that it is satisfying our “intent”. And if operant conditioning does influence the way we behave, maybe we do actually need to start thinking about what the machine algorithms are training us to do. Are training us to do. Training us.

Google’s stated aim is to “organize the world’s information and make it universally accessible and useful”.

- Through web search, it started to organise information it presented to use through search results that were more appealingly ranked (seemed “more relevant”) than the other search engines did.

- Through personalised search, it started to organise the way it presented results to each of us individually.

- Through web tracking, it presents us with information – adverts – organised in a way it presumably thinks are more personally meaningful to use (but maximising what metic exactly? More likely to cause us to act in a particular way, as measured by whether we click the link, or linger on a page, or engage in a particular behaviour that can be captured – for model building and exploitation purposes – by web tracking algorithms?)

- Through Google Now, and the new Google image gallery tools, Google is seeking to organise our information (we’re part of the world, right?) on our behalf and present it back to us in a way that the Google algorithms decide.

The old photos in a drawer back at my family home are sorted howsoever (by whatever algorithm “use” and random access results in). Now they’ll be sorted by Google. Maybe the algorithms are similar. Or maybe they’re not. What would be evil, I think, was if the ranking algorithms that are used to decide the order in which organic information is presented us start to be influenced by the algorithms that are tied to advertising or marketing, that is, to algorithms that are used to try to maximise the extent to which we are influenced in accord with the goals, beliefs, desires and intents of others (with a hat tip there to agent logic and the theories of intelligent software agents).

At the moment I believe that Google believes it is trying to develop algorithms that benefit us personally, in an utilitarian way. But I’m not sure what function it is they are maximising or how they think it maps onto any personal theories or preferences we may have about what is “accessible” and “useful”. I guess we might also ask whether “accessible” and “useful” are the road to a Good Life (because in the end this comes down to philosophy and ethics, doesn’t it?) or whether we should be “organising the world’s information” with some other purpose in mind?

PS Just by the by, it’s worth noting that the educational arena is seeking to use learning analytics to instrumentalise our behaviour and engagement within learning systems and contexts for our, erm, learning benefit. (Measured how?)

And as anyone following my tweets over the last year or two will know, I’ve started finding Google more and more irksome.

It’s not just that the one or two people I know who use Google Plus (Google+?) are now all but lost to me as sources of neat ideas because I don’t do Gooplus and it doesn’t do RSS…

It’s not just because Google is shutting down the Google Reader backbone that powers a lot of RSS and Atom syndication feed services (and leaves me wondering: how long is Feedburner for this world? Maybe it’s time to start moving your feeds and trying to get folk off that piece of infrastructure…)…

It’s not just that geocoding done within Fusion Tables is not exported – if you look at a KML feed from Google Fusion Tables, you’ll find there’s no lat-long data there. To get a geo-view, you need to stick in Google Fusion Tables or wire the feed into Google Earth, which will then “initiate geocoding of location descriptions while viewing [the] KML file”…

It’s not just that Google is deprecating gadgets from spreadsheets, which as Martin points out means that if I want to visualise data in a spreadsheet all I’m going to be left with is Google’s crappy charts…

It’s not just that Google moved away from using CalDav to support calendar interoperability… (announcement: “CalDAV API will become available for whitelisted developers, and will be shut down for other developers on September 16, 2013. Most developers’ use cases are handled well by Google Calendar API, which we recommend using instead.”)

It’s not just that Google is moving away from using the XMPP instant messaging protocol (and nor, I think, making a move towards using MQTT?)…

It’s not just that Google will be using your photos to create photos you never took and presumably offer them up via your image gallery in favour of photos it thinks aren’t up to scratch…

Though I’m sure that Google wouldn’t start pushing images in just the WebP image format so that you’d feel obliged to use Chrome…

And also in the browser, I’m sure Google wouldn’t start using Google Public DNS as a Chrome default setting. (Is the same true of Chromebook? Presumably folk connected to Google Fiber use Google Public DNS?) But does it use SPDY as a default? How about on Android?

It’s not just that Google will tag your social media posts using tags you might never use yourself, and as it does so altering the externalised memory embodied by that post…

It’s not just that as web search gets increasingly personalised and localised, we lose any sense of Google ground truth; I’m not quite sure how the info-skills trainers are going to address this when training a motley crew of different learners to discover a particular resource other than by using known-item search strategies (which sort of misses the point). Or maybe it’s right that a cohort of students should all get different results when they run ostensibly the same search?

Hmmm.. thinks: if personalised/localised search could be reduced to raw search phrase (whatever I put in the search box) plus a set of invisible search limits that reflect the personalisation/localisation tweaks applied to my search, how might my hidden/invisible search limits compare with yours?

It’s not just that Google uses tax efficient corporate structures to minimise its tax bill, because lots of companies do that…

It’s not just any one of these things, taken on its own merits… it’s all of them taken together…

“Embrace, extend, extinguish”… where have we heard that before?

Drip; drip; drip…

PS see also M. Wunsch on The Great Google Goat Rodeo

PPS Although not an open standard, I forgot this one – Google dropped support for the closed Microsoft ActiveSync protocol (see also Google Sync End of Life)

Lincolnshire

The Lincolnite – Lincolnshire Local Elections 2013, described here: The Lincolnite to cover elections live with interactive map. (Ex-?) University of Lincoln developer Alex Bilbie (@alexbilbie), who built the Lincolnshire map app, describes a little of the process behind it here Developing an interactive county council election map (part one).

Warwickshire

2013 Elections: Warwickshire area, described here: 2013 Elections – In Real Time.

The Warwickshire team are also making shapefiles/KML files (for the plotting of boundary line maps) and live results data (via Google Fusion Tables) too, as well as making data available about previously elected candidates: 2013 Elections – In Real Time

Here’s the map after the fact… I like the clear statement of seats by part in the bottom left corner too…

Surrey

Surrey has the Surrey elections dashboard (via @BenUnsworth) that will switch to a live map as the results come in, but currently offers a search box that accepts a postcode and then tells you who your candidates are and where you can vote:

Kent
This looks pretty, from Kent County Council:

I was looking forward to seeing how this view played out once the results started to come in, but, erm, oops?!

Managing the bursty load on an election results service server is probably something worth building into the planning…

Bristol
Bristol opted for a tabbed display showing a range of views over their council elecitons results. A simple coloured symbol maps shows the distribution of seats by ward across the parties:

Bristol also provided a view over the turnout:

(Would it be useful to also be able to see this as percentage turnouts? Or to depict the proportional turnout on a map to see if any geographical reasons jump out as a source of possible differences?)

Cumbria
Cumbria County Council show how to make use of boundary files to mark or <choropleth election maps relate the party affiliation of the candidate taking each particular seat by electoral division area:

Cumbria also provided a view of seat allocations in districts; I don’t understand the scale they used to on the x-axes though? It differs from district to district. For example, it looks to me as if more seats went to Conservatives in Eden than in Carlise? Or is the scale related to the percentage of seats in the district? I’d class as “infographic-standard”, i.e. meaningless as a visualisation;-)

Norfolk

Norfolk’s election map looks quite, erm, “child-friendly” (chunky?! kids TV?) to me?

Norfolk also produced a graphic showing how seats might be distributed in the chamber:

I think one of the major issues with this sort of graphic is how you communicate the possible structurings of the chamber based on what sort of affiliations and groupings play out?

Wales Online

Wales online have a nice clean feel to their results map for Anglesey, but what’s going on with the legend? They don’t make it easy to get the branding into the screengrab either?!

National Reporting

THe Telegraph produced a map showing results of the elections at national scale based on control of councils by party:

And ever helpful, the Guardian datablog made the data available (will they do data broken down at seat level too, I wonder?) Here’s the map they produced:

Hmmm… do I recognise that sort of layout? Ah, I know, it reminds me of this example of Data Journalists Engaging in Co-Innovation… around boundary changes.

Other…
For lists of current councillors, see OpenlyLocal, which has data available via a JSON API relating to current councillors and their affiliations. (It would be good if a frozen snapshot of this could be grabbed today, for comparison with the results following today’s election?)

This may also be of interest… UK Data Service British Local Election Database, 1889-2003 and Andrew Teale’s Local Elections Archive Project.

Data relating to general elections can be found on the Electoral Commission website: General Elections. TheyWorkForYou provide an API over current MPs by constituency, and MySOciety also produce the MapIt service for accessing constituency and electoral division boundary line data files.

If you’re interested in doing something data related around the election, or would like to learn how to do something with the data generated by the election, check out this informal resource co-ordination document. If you’re interested in checking out your local council website to see whether they publish any #opendata that would help you generate you own live maps, dashboards or consoles, the School of Data post “wot I wrote” on Proving the Data – A Quick Guide to Mapping England and Wales Local Elections may provide you with a quick start guide to making use of some of it…

If you know of any other councils or local presses publishing election related data warez, maps, live data feeds, etc, please post a link and brief description in the comments, and I’ll try to keep this post up to date…

Another source of data is MySociety’s MapIt service, which provides data about various geographies, including electoral wards, covered by other geographical areas:

as well as boundary files for each geography in a variety of formats:

Unfortunately, the MapIt API doesn’t (yet?!) support the ability to generate a single file that contains boundary data for all the electoral wards in a single council or local authority area. So here’s a quick hack, posted as a view on Scraperwiki, that generates a single KML file for the electoral wards contained by a council area (including the wider boundary of that council area) – KML Merge Test. (There are probably better ways of doing this?! I would if I should try to make sense of the MapIt code to see if I can work out how to submit a proper patch…)

import scraperwiki,simplejson,urllib2
from lxml import etree
from copy import deepcopy


#--via @mhawksey
# query string crib https://views.scraperwiki.com/run/python_querystring_cheat_sheet/?
import cgi, os
qstring=os.getenv("QUERY_STRING")

key='65791' #Use the Isle of Wight as a default
typ='UTE'
#typs are:
#CTY (county council), CED (county ward), COI (Isles of Scilly), COP (Isles of Scilly parish), CPC (civil parish/community), CPW (civil parish/community ward), DIS (district council), DIW (district ward), EUR (Euro region), GLA (London Assembly), LAC (London Assembly constituency), LBO (London borough), LBW (London ward), LGD (NI council), LGE (NI electoral area), LGW (NI ward), MTD (Metropolitan district), MTW (Metropolitan ward), NIE (NI Assembly constituency), OLF (Lower Layer Super Output Area, Full), OLG (Lower Layer Super Output Area, Generalised), OMF (Middle Layer Super Output Area, Full), OMG (Middle Layer Super Output Area, Generalised), SPC (Scottish Parliament constituency), SPE (Scottish Parliament region), UTA (Unitary authority), UTE (Unitary authority electoral division), UTW (Unitary authority ward), WAC (Welsh Assembly constituency), WAE (Welsh Assembly region), WMC (UK Parliamentary constituency)

if qstring!=None:
    get = dict(cgi.parse_qsl(qstring))
    if 'key' in get: key=get['key']
    if 'typ' in get: typ=get['typ'] 
#---


#Get a stub KML file for the local council level
url='http://mapit.mysociety.org/area/'+str(key)+'.kml'
xmlraw = urllib2.urlopen(url).read()
xml=etree.fromstring(xmlraw)

#Get the list of electoral wards covered by that council area
wards=simplejson.load(urllib2.urlopen('http://mapit.mysociety.org/area/'+str(key)+'/covers?type='+typ))

#Get the KML for each ward, extract the Placemark data, and add it to our comprehensive KML tree
for ward in wards:
    url='http://mapit.mysociety.org/area/'+ward+'.kml'
    xmlraw = scraperwiki.scrape(url)
    xml2=etree.fromstring(xmlraw)
    p= xml2.xpath('//geo:Placemark',namespaces={'geo':'http://www.opengis.net/kml/2.2'})
    xml.append( deepcopy(p[0] ))

scraperwiki.utils.httpresponseheader("Content-Type", "text/xml")
print etree.tostring(xml)

The key value is the ID of the council area within which you want to find the electoral wards. So for example the Isle of Wight parliamentary constituency page -http://mapit.mysociety.org/area/65791.html – gives us the ID 65791, which we pass as an argument to the Scraperwiki view. The subdivision we want to grab data for is given by the typ parameter:

CTY (county council), CED (county ward), COI (Isles of Scilly), COP (Isles of Scilly parish), CPC (civil parish/community), CPW (civil parish/community ward), DIS (district council), DIW (district ward), EUR (Euro region), GLA (London Assembly), LAC (London Assembly constituency), LBO (London borough), LBW (London ward), LGD (NI council), LGE (NI electoral area), LGW (NI ward), MTD (Metropolitan district), MTW (Metropolitan ward), NIE (NI Assembly constituency), OLF (Lower Layer Super Output Area, Full), OLG (Lower Layer Super Output Area, Generalised), OMF (Middle Layer Super Output Area, Full), OMG (Middle Layer Super Output Area, Generalised), SPC (Scottish Parliament constituency), SPE (Scottish Parliament region), UTA (Unitary authority), UTE (Unitary authority electoral division), UTW (Unitary authority ward), WAC (Welsh Assembly constituency), WAE (Welsh Assembly region), WMC (UK Parliamentary constituency)

So for example, here’s a link to an aggregate KML file for Unitary authority electoral divisions (UTE) on the Isle of Wight – https://views.scraperwiki.com/run/kml_merge_test/?key=65791; and here’s one for Unitary authority wards (UTW) in Milton Keynes: https://views.scraperwiki.com/run/kml_merge_test/?key=2588&typ=UTW

If you save the resulting file as a .kml file (for example, as kml_merge_test_mk.kml) you can then load it into something like Google Fusion tables to view it:

Note that the MapIt API is rate limited (I think), so be gentle ;-)

A post_type attribute identifies whether an entry is a forum, topic or reply.

I grabbed three columns of data defining an edge list – parentID, postID, post_type (forum, topic or reply), and threw the data into Gephi. Different layout types bring out different structural elements, but I just wanted something quick and dirty to look at the data in “unthreaded” form to see how we might make use of the thread/menu_order data.

Here’s an example – the whole structure (not very compellingly laid out, it has to be said):

We can also filter by edge type to get a hint of the actual structure, for example, the forum structure (i.e. how forums and subforums (child forums) relate to each other):

Or the topic structure (how topics relate to forums):

Or the reply structure (how replies fall into topics):

Note that this is the one we need to work on, in respect of finessing the data to allow us to see the hierarchical nature of the thread structure within a topic.

Anyway, as I said, a shot to nothing, and something to ponder representation wise as I walk the dog…;-)

This casual thinking is compounded both by working in the education sector, and by the “personal research: label I apply to much of what I do. That is, I don’t work for a private company to use these tools to produce something that will be sold for profit.

So when I read Martin Belam’s review of the FT’s Emily Cadnam’s thoughts on data journalism in a news:rewired panel last week, I was bumped into rethinking things:

Along the way she made a little point with a big implication, certainly one that doesn’t always occur to me. Because the FT is a paid for product, they are actually restricted from using a lot of open source and free tools from the web, which have licenses that forbid commercial use. As an unintended consequence of having a website with a subscription model, they’ve had to make their own versions of several fundamental tools that others might take for granted.

Hmmm… so which of the tools and techniques that I try to advocate are actually closed for journalistic use in commercial contexts?

PS Thinks: when I download and use a software tool, accepting terms and license of use conditions as I do so, under what legal framework does the tool publisher claim ownership of the software application and the right to issue and enforce the license conditions or terms of use? Copyright applies to the source code, and I guess the compiled code? So to the extent that a musician may own copyeight over an expression (recording) of a song, maybe a software publisher owns copyright in the executable version of a piece of software, and running that application is like listening to a piece of recorded music. And just as you are limited in the extent to which you can play recorded music (not to a large audience, for example?) presumably similar conditions can be made to apply to the use to which you put a piece of software? (See for example, The Closed Route to Open Data.)

If the route is by copyright, then is there any notion of a fair dealing exception in the use of tools for the purpose of reporting the news?! (A ludicrous thought, but I thought I’d capture it anyway…;-)

PPS Hmmm, I wonder.. I can imagine patent trolls thinking this through…:

• teach a man to fish, get a royalty payment each time he catches one;
• sell a man a fishing rod, get a royalty payment each time he catches a fish;
• sell a man a fishing rod, get a royalty payment each time he uses it;
• teach a man the idea of fishing, get a royalty payment every time he catches a fish;
• teach a man the idea of fishing, get a royalty payment every time he catches anything (use of idea in derived work..);

There must be an edu-startup killer trollable patent in there somewhere?!;-)

The story so far: an academic paper used to bolster austerity policy arguments was shown to include errors, frame the data in a particular way, and make a hardline point around an arbitrary threshold value, as for example described in The 90% question [The Economist] or Reinhart-Rogoff Revisited: Why We Need Open Data in Economics [Open Economics blog]. THe story of the graduate student project that led to the mistakes being discovered is described in the BBC News Magazine: Reinhart, Rogoff… and Herndon: The student who caught out the profs.

Here’s the BBC’s “More Or Less” take on it (personal archive copy) :

Some observations on related matters:

• if you ever work with data, you’ll know that you have to be selective about what data you include in analysis; sometimes data isn’t available, sometimes it comes from different sources that use slightly different interpretations or definitions, sometimes you throw data away because it clutters a storyline you are trying to explore. For me, data analysis is a conversation with a data. The conversation is caveated and explores a particular issue, “all other things being equal”, which they often aren’t; but we build that into our caveats as we tell stories to ourselves. Caveats and qualifiers and particular rationales which don’t get communicated to a wider audience, maybe, or the consequences of which aren’t fully appreciated by a wider audience. Although they are by the analyst/researcher. (“You took it out of context”, “but it was obviously a joke”, etc etc). You can look at a sculpture from many different directions. Same with data.
• things get picked up by official reports and become “truth”, sometimes losing their flaky or biased provenance as they get “washed” though being cited in more and more “weighty” reports or publications (eg Sleight of Hand and Data Laundering in Evidence Based Policy Making). Others become zombie statistics, refusing to die and repeatedly being cited as evidence, even if they have been debunked.
• I used to share an office with psychology researchers, whose day was made if they had an experimental result “significant at the 5% level”, crushed if the significance level was outside it. (What are good references containing arguments against null hypothesis significance testing?). Tell yourself a story, then make the judgement. Hard threshold values can make for dodgy decisions (one reason I like the notions of fuzzy logic and hysteresis). If your results have accuracy bounds, rerun the numbers a few times with different errors that take into account those tolerances. Can you make an insignificant result significant, at least once, and then quote that? #FFS There’s no truth there…Or maybe there are lots of sort-of truths? (See also: Ben Goldacre’s Bad Pharma, especially the chapters on “Missing Data” and “Bad Trials”.)
• spreadsheets can be really hard to debug. I don’t think they have a “View Source” option that just shows all the formulae and outline’s there ranges, as well as maybe highlight fixed cells that have been overwritten into otherwise calculated cell ranges, do they? See also: EuSRIG – the European Spreadsheet Risks Interest Group, which I guess has come to many folks’ attention for the first time over the last week, and whose summer conference enrolment numbers may well take a leap, not least from press access requests?!
• there are shed loads of papers out there in the “peer reviewed” literature, though that’s not to say the peer reviewers actually tried to replicate the results.
• although this has slipped onto the backburner a bit (where does the time go?!:-(, I tried to frame my own personal learning journey into the world of statistics around replicating various academic papers about motor sport (and in particular, Formula One) results and timing data. I’ve only managed to attack one so far (here and here), but already it through up some interesting observations about the data that was used to generate correlations, such as whether you rank cars that weren’t classified. The used of significance tests also seemed a bit – pointless – to me. Anyway – the point is this: there are a load of papers out there on a whole range of topics that might provide a good basis for “textbook examples” to help folk learn how to use particular analysis tools or techniques. As well as describing the method, (supposedly!), many papers include “answers” that you can use to check your working (or theirs!). In much the same way that the School of Data is trying to develop the idea of “Data Expedition” style uncourses, where folk come together to find, analyse and tell stories from datasets in particular topic areas, how about a notMOOC pedagogical style based around working through and replicating the findings of particular published academic papers, which might also involve learning precursor stuff that you need to know in order to make sense of the paper or try out its analyses?

Hmmm…

To pull the post together, I drew on a Guardian Technology editor Charles Arthur’s retweets over the course of the evening, which provided an interesting technological slant on the news as it was breaking/developing.

The event also prompted a series of posts on the role of Twitter in reporting breaking news. Twitter and news: The canary down the mine compares the dynamics of Twitter in a breaking news situation in the context of the “news funnel”:

One of the theories you learn when you train to be a journalist is the “funnel” of news. Imagine a funnel. It’s getting all the information about a certain news story poured into it – from the top. Wild rumours and hard facts. Witness accounts alongside back-of-a-cigarette packet theories,

The funnel is the journalist. And the funnel’s job is to take all the information, from the crazy and the correct, and pour it, with a measure of control, into the story. Take out the impurities, crush up the lumps, and make the resulting article a distillation of the thousands of snippets, with no errors.

It seems to me, very often these days, that Twitter is the funnel turned upside down.

All of those disparate ingredients are poured into the nozzle. The narrow part. And they are not filtered, not regulated, not tested. But they come out of the wide part of the funnel. They are spread across a large area. Indiscriminately.

And that’s how Twitter appeared to me on the night of the Boston marathon bombs.

Another review of the event used a form of topic analysis to explore the semantic textual content different waves of Twitter activity using a Twitter accession count timebase – The evolution of discussion around the Boston Marathon events.

PS in the context of Twitter and the news, I note that Guardian Datablog editor Simon Rogers is going to work for Twitter, to be replaced at the Guardian by James Ball… I also spotted a few days ago that ex-of the Guardain data intereactive designer Alastair Dant had also moved to the New York Times.

and had a beer:

Very – apt.

I’m normally a bitter drinker (read that how you will…) but when in, erm, Rome, I figured I should do as the, erm, doughnuts do, and have what presumably translates back here as “a continental beer”:

Good trip though:-)