In his book Visualising Data: Exploring and Explaining Data with the Processing Environment, Ben Fry describes a seven stage process for understanding data:

Acquire
Obtain the data, whether from a file on disk or a source over a network.
Parse
Provide some structure for the data’s meaning, and order it into categories.
Filter
Remove all but the data of interest.
Mine
Apply methods from statistics or data mining as a way to discern patterns or place the data in a mathematical context.
Represent
Choose a basic visual model, such as a bar graph, list or tree.
Refine
Improve the basic representation to make it clearer and more visually engaging.
Interact
Add methods for manipulating the data or controlling what features are visible.

I’m not sure I’d agree with the elements above defining a rigid linear process

(Produced using Graphviz; dot file)

I prefer to think of the way I work as something like this:

(Produced using Graphviz; dot file)

but what is clear is that we need to understand what data is available to us.

As I mentioned in F1 Data Junkie – Getting Started, the data I’m going to be playing with (at least at first) is data grabbed from the Mclaren F1 Live Dashboard (as developed by Work Club) so let’s have a look at it… (I’ll come back to how the data was acquired in a later post.)

The data – which I think is updated on the server at a rate of once per second (maybe someone form Work Club could confirm that?) – is published as JSON (Javascript Object Notation) data in a callback function (so it uses what is referred to as the JSON-P (“JSON with Padding”) convention. What this means is that the dashboard web page can call the server and get a some data back in such a way that as soon as the data is loaded into the page, a Javascript programme function can run using the data that has just been downloaded).

Here’s what the data as grabbed from the server looks like:

Dashboard.jsonCallback('{
 "drivers":{
  "HAM":{
   "code":"HAM",
   "telemetry" {
    "timestamp":"15:40:58.383",
    "nEngine":13920,
    "NGear":2,
    "rThrottlePedal":68,
    "pBrakeF":2,
    "gLat":1,
    "gLong":1,
    "sLap":3116,
    "vCar":96.9,
    "NGPSLatitude":26.03185,
    "NGPSLongitude":50.51304
   },
   "additional":{
    "lap":"17",
    "position":"5",
    "is_racing":"1"
   }
  },
  "BUT":{"code":"BUT","telemetry":{"timestamp":"15:40:58.790","nEngine":13106,"NGear":3,"rThrottlePedal":49,"pBrakeF":2,"gLat":2,"gLong":0,"sLap":2681,"vCar":140.2,"NGPSLatitude":26.0333,"NGPSLongitude":50.51635},"additional":{"lap":"17","position":"8","is_racing":"1"}}},
  "commentary":[
   {
    "name":"COM",
    "initials":"CM",
    "text":"Lewis sets the fastest lap with a 2\'00:447 that time around.",
    "timestamp":"2010-03-14 12:40:14"
   }
  ]
 }
');

So what data is there? Well, there are telemetry data fields for each driver:

• timestamp – the time of day;
• nEngine – the number of revs the engine is doing;
• NGear – the gear the car is in (over the range 1 to 7);
• rThrottlePedal – the amount of throttle depression(?) (as a percentage);
• pBrakeF – the amount of brake depression(?) (as a percentage)
• gLat – the lateral “g-force” (that is, the side-to-side g-force that you feel in a car when going round a corner too quickly);
• gLong – the longitudinal “g-force” (that is, the forwards and backwards force you feel in a car that accelerates or decelerates quickly when going in a straight line);
• sLap – the distance round the lap (in metres); this resets to zero on each tour, presumably at the start/finish line);
• vCar – the speed of the car (km/h);
• NGPSLatitude – the GPS identified latitude of the car;
• NGPSLongitude – the GOS identified longitude of the car.

On some samples, there is also commentary information, but I’m going to largely ignore that..

Parsing

The data I got hold of was a bundle of files containing data in the JSONP format like that shown above, with one file containing one package of data created once a second.

In order to parse the data, I needed to decide what format I wanted it in for processing. The format I chose was CSV – comma separated variable data – that looks like this:

The first column is the original filename, the other columns correspond to data downloaded from the Mclaren site.

In order to generate the CSV data, I wrote a Python script that would:
- strip off the padding around the JSON data;
- parse the JSON using a standard Python JSON parsing library;
- (add a line to strip out escape characters that weren’t handled correctly by the parser and place it before the parsing step);
- use a CSV library to write out the data in the CSV format.

(See an example Python script here.)

I then refined my parsing script so that it would generate one CSV file per lap. To do this, the script had to:
- detect when the lap distance in one sample was less than the distance in the previous sample (i.e. the lap distance measure has been reset to zero between the two samples, using a construction of the form if oldDist > d['sLap']: where oldDist = d['sLap'] once we have written the corresponding data record to the CSV file);
- if a new lap has been started, close the old CSV file, create a new one, write the column header information into the top of the file, and then start adding the data to that file.

Having got the data into a CSV format, I could then load it in to an environment where I could start to think about visualising it. A spreadsheet for example, or Processing, (which is what I used to create the single lap view shown at the start of this post).

But to see how do that on the one hand, and what stories we can find in the data on the other, we’ll need to move on to another post…

[Reflection on this post: to get a large number of folk interested, I really need to do less of the geeky techie stuff, and more of the "what does the data say" cool viz stuff... but if I don't log the bootstrap techie stuff now before I overcomplicate it(?!) my record of the simplest file handling and parsing code will get lost...;-) In the book version, a large part of this post would be a technical appendix... But the "what data fields are available" bit would be in Chapter 2 (after a fluffy Chapter 1!).]

PS some of the technical details behind the Mclaren site have started appearing on the personal blog of one of the developers – e.g. Building McLaren.com – Part 3: Reading Telemetry. In that post it was pointed out I haven’t been adding copyright notices about the data to the posts – which I’ll happily do once I know who to acknowledge, and how… In the meantime, it appears that “the speed, throttle and brake are sponsored by Vodafone” and “McLaren are providing this data for you to view” so I should link to them: thanks McLaren :-)