# Narrative Charts Tell the Tale…

A couple of days ago, I got a message from @fantasticlfe asking if I’d done any tinkerings around what turned out to be “narrative charts”. I kept misapprehending what he was after (something to do with continuity?!;-), so here’s a summary of various graphical devices for looking at narrative texts that we passed back and forth, along with some we didn’t..

A Sankey diagram typically uses variable thickness lines to show flow between different elements in a system. (For this reason it’s often used to show energy flows throuygh a system, though it can also be used to good effect to show money flows.) The chart Michael linked to comes from xkcd:

In this chart, we have time along the horizontal x-axis. The y-axis is ambiguous (some sort of nominal ordering?) and the line thickness appears to represent army size.

To a certain extent, this diagram is reminiscent of Minard’s famous chart…

(See also What Makes a Minard? for some contemporary Minard diagrams. Is code available, I wonder?)

However, in the case of Minard’s chart (which I personally don’t like at all!), the x-y and co-ordinates represent map co-ordinates – the thick lines aren’t thick lines in a line chart (which a glanced “up and to the right” view might make you assume), they’re flow lines across a map.

I got distracted for a while by the Sankey aspect, and dug around my own bits of code. For example, Generating Sankey Diagrams from rCharts, an rCharts wrapper for the d3.js Sankey diagram. Michael was particularly interested in being able to group lines vertically (though I wasn’t sure what the y-axis would actually correspond to: some loose function of “location”, maybe as a categorical variable? Time was definitely to be on the horizontal x-axis); a posting on Stack Overflow (d3 sankey charts – manually position node along x axis) seemed likely to be able to help with that.

I then started going off on one…

Would a variant of nltk style lexical dispersion plots help, using characters rather than word categories? That would show when a character was in scene, but not much else?

How about sentence drawing, in which we show “turns” taken by different speakers?

This shows something, but again, not relevant…

Nor are Kurt Vonnegut’s shapes-of-stories diagrams that plot some sort of emotional state on y and time on x:

Hmmm… Michael wanted to be able to look at scenes on x and presumably some function of location on y. Hmm… why? And how might we actually order those axes? Scenes occur in order in a film or play, but scene is a ranked, ordinal value. That said, scenes also have duration in terms of screentime, which may or may not be the same as the “interval” that the scene portrays in terms of the world it represents (this must have a name? eg a 20 second screen time scene shows a plane flying and this represents x hours in the story). The scene may also have a ‘calendar time’ associated with it in the story – so where you have a flashback scene this corresponds to a previous calendar time in the represented world. Did Michael want any of these dimensions capturing?

Related to shapes of stories, here’s how someone analysed several thousand plots: Examining the arc of 100,000 stories: a tidy analysis.

And then there’s location… how should these be represented? Locations are a distance apart and, perhaps more importantly from a continuity point of view, a travel time apart; as well as maybe a timezone difference apart. Did that need capturing in any way? Ordering axes for this could be quite hard if we wanted close things in space (distance? travel time?) to be close together on a single axis (A is 10 minutes from B and C, B is ten minutes from C: how do you show that intransitive relation on a single dimension? [Maybe relevant? Storygraph: Extracting patterns from spatio-temporal data, A Shrestha et al., Advances in Visual Computing.] Hmm… If we can capture distance between locations, and some sensible notion of time relating to scenes, could we maybe use line thickness to show that a person has lots of time to move between one (time, location) and another, as compared to scenetime? Do filmwriters have tools to support this? Do the police…?! Is the Mythology Engine relevant?

How about thinking about it as a graph? I’ve used Gephi before as a foil for getting me to think about ordered series as connected events in a graph – for example, Visualising F1 Timing Sheet Data. If we encode scene number as the x-coordinate and location number as the y-coordinate, with each graph line being the connected series of scenes a particular individual is in, then we can simply use a line chart to connect “individual lines” to different scene and location numbers. We’d also have a couple of extra dimensions to play with – node size and node colour, at each location. We’d also have the opportunity to play with edge (that is, line) colour and edge thickness?

Maybe I need to try to do some demos? But no time for that right now…

How about trying to find some? Here are some discovered via @jamesjefferies:

Here’s a view of connected (by travel between) locations in Game of Thrones:

There’s also an animation of event in Game of Thrones, but I can’t quite figure out how to read it?!

Let’s go back to the sort of thing Michael was after – narrative charts..

@imhelenj found a related if cluttered interactive describing the evolution of web tech:

Then Michael shared a link to Comic Book Narrative Charts, a project for automatically generating xkcd style narrative charts:

Hovering over these charts, I noticed they were interactive d3.js charts. A quick View Source and the code for generating the chart dynamically from a characters file and a narrative file appeared to be there. Which I think is what Michael wanted all along…!

(By the by, the post also describes how the developers started thinking about fixing the vertical y-coordinate values. Here’s another example of someone thinking aloud around producing a narrative chart for the Holy Week story.)

Ho hum, an interesting set of detours nonetheless – and it got me thinking about the time-space complexity of a scene based tale that could keep be confused for weeks! :-)

PS this is quite interesting – visualising a process, via Tactical Tech Drawing By Numbers project:

PPS some more bits: @r4isstatic points to Some visualisations of stories and narratives, another summary post similar to this one. Also via Paul Rissen, and picking up on whether the police have any interesting actor/event/time/location diagramming techniques, Vispol – An Interactive Scenario Visualization.

Elsewhere, I find Storyline Visualizations, which includes a paper (Design Considerations for Optimizing Storyline Visualizations, Y Tanahashi, and K-L Ma, IEEE Trans on Visualisation and Computer Graphics, 18(12) 2012, pp2679-2688 and some python code.

PPPS Some more… A collection by Stewart McKie of techniques for visualising screenplays: Screenplay Visualization: Concepts and Practice. The posts I wrote on the Digital Worlds game design uncourse blog about narrative structure. Sort of via Scott Wilson, some crime analysis software from xanalys.com (Link Explorer – White Paper) which includes descriptions of an event chart, a transaction chart and an activity timeline:

Via the comments, this rather lovely animated discourse map:

# Twitter Audience Profiling – OU/BBC Feynman Challenger Co-Pro

Another strong piece of TV commissioning via the Open University Open Media Unit (OMU) aired this week in the guide of The Challenger, a drama documentary telling the tale of Richard Feynman’s role in the accident enquiry around the space shuttle Challenger disaster. (OMU also produced an ethical game if you want to try you own hand out at leading an ethics investigation.)

Running a quick search for tweets containing the terms feynman challenger to generate a list of names of Twitter users commenting around the programme, I grabbed a sample of their friends (max 197 per person) and then plotted the commonly followed accounts within that sample.

If you treat this image as a map, you can see regions where the accounts are (broadly) related by topic or interest category. What regions can you see?! (For more on this technique, see Communities and Connections: Social Interest Mapping.)

I also ran a search for tweets containing bbc2 challenger:

Let’s peek in to some of the regions…”Space” related twitter accounts for example:

Or news media:

(from which we might conclude that the audience was also a Radio 4 audience?!;-)

How about a search on bbc2 feynman?

Again, we see distinct regions. As with the other maps, the programme audience also seems to have an interest in following popular science writers:

Interesting? Possibly – the maps provide a quick profile of the audience, and maybe confirm its the sort of audience we might have expected. Notable perhaps are the prominence of Brian Cox and Dara O’Briain, who’ve also featured heavily in BBC science programming. Around the edges, we also see what sorts of comedy or entertainment talent appear to the audience – no surprises to see David Mitchell, Charlton Brooker and Aianucci in there, though I wouldn’t necessarily have factored in Eddie Izzard (though we’d need to look at “proper” baseline interest levels of general audiences to see whether any of these comedians are over-represented in these samples compared to commonly followed folk in a “random” sample of UK TV watchers on Twitter. The patterns of following may be “generally true” rather than highlighting folk atypically followed by this audience.)

Useful? Who knows…?!

(I have PDF versions of the full plots if anyone wants copies…)

# BBC Question Time “Extra Guest” – On Twitter…

For some time, BBC Question Time has benefitted from an active online live commentariat via SMS/text messages displayed on a lower third ticker, and on on Twitter around the #bbcqt hashtag.

Now, via @federicacocco, I notice that BBC Question Time has a new Twitter account associated with it, @BBCExtraGuest:

Here’s some sort of confirmation, from the official @bbcquestiontime account, that it’s legitimate:

The “extra guest” twist that BBC Question Time are introducing is one that maybe we can also adopt at the OU? @OUAcademic, maybe, a rotating “guest” slot in which a nominated academic can tweet along to a live broadcast of an OU/BBC co-pro?

PS to keep track of OU/BBC co-pros, see:

PPS for a quick sketch comparing the folk commonly followed by the users of the #bbcqt and #newsnight hashtags, (that is, an approximation of the common and different social interests of the audiences of those two programmes) see Social Media Interest Maps of Newsnight and BBCQT Twitterers.

# Feeding on OU/BBC Co-Produced Content (Upcoming and Currently Available on iPlayer)

What feeds are available listing upcoming broadcasts of OU/BBC co-produced material or programmes currently available on iPlayer?

One of the things I’ve been pondering with respect to my OU/BBC programmes currently on iPlayer demo and OU/BBC co-pros upcoming on iPlayer (code) is how to start linking effectively across from programmes to Open University educational resources.

Chatting with KMi’s Mathieu d’Aquin a few days ago, he mentioned KMi were looking at ways of automating the creation of relevant semantic linkage that could be used to provide linkage between BBC programmes and OU content and maybe feed into the the BBC’s dynamic semantic publishing workflow.

In the context of OU and BBC programmes, one high level hook is the course code. Although I don’t think these feeds are widely promoted as a live service yet, I did see a preview(?) of an OU/BBC co-pro series feed that includes linkage options such as related course code (one only? Or does the schema allow for more than one linked course?) and OU nominated academic (one only? Or does the schema allow for more than one linked academic? More than one), as well as some subject terms and the sponsoring Faculty:

```  <item>
<title><![CDATA[OU on the BBC: Symphony]]></title>
<description><![CDATA[Explore the secrets of the symphony, the highest form of expression of Western classical music]]></description>
<image title="The Berrill Building">http://www.open.edu/openlearn/files/ole/ole_images/general_images/ou_ats.jpg</image>
<bbc_programme_page_code>b016vgw7</bbc_programme_page_code>
<ou_faculty_reference>Music Department</ou_faculty_reference>
<ou_course_code>A179</ou_course_code>
<transmissions>
<transmission>
<showdate>21:00:00 24/11/2011</showdate>
<location><![CDATA[BBC Four]]></location>
</transmission>
<transmission>
<showdate>19:30:00 16/03/2012</showdate>
<location><![CDATA[BBC Four]]></location>
</transmission>
<transmission>
<showdate>03:00:00 17/03/2012</showdate>
<location><![CDATA[BBC Four]]></location>
</transmission>
<transmission>
<showdate>19:30:00 23/03/2012</showdate>
<location><![CDATA[BBC Four]]></location>
</transmission>
<transmission>
<showdate>03:00:00 24/03/2012</showdate>
<location><![CDATA[BBC Four]]></location>
</transmission>
</transmissions>
<category domain="http://www.open.edu/openlearn/whats-on">What's On</category>
<category domain="http://www.open.edu/openlearn/tags/bbc-four">BBC Four</category>
<category domain="http://www.open.edu/openlearn/tags/music">music</category>
<category domain="http://www.open.edu/openlearn/tags/symphony">symphony</category>
<pubDate>Tue, 18 Oct 2011 10:38:03 +0000</pubDate>
<dc:creator>mc23488</dc:creator>
</item>```

I’m not sure what the guid is? Nor do there seem to be slots for links to related OpenLearn resources other than the top link element? However, the course code does provide a way into course related educational resources via data.open.ac.uk, the nominated academic link may provide a route to associated research interests (for example, via ORO, the OU open research repository), the BBC programme code provides a route in to the BBC programme metadata, and the category tags may provide other linkage somewhere depending on what vocabulary gets used for specifying categories!

I guess I need to build myself a little demo to se what we can do with a fed of this sort..?!;-)

I’m not sure if plans are similarly afoot to publish BBC programme metadata actual the actual programme instance (“episode”) level? It’s good to see that the OpenLearn What’s On feed has been tidied up little to include title elements, although it’s still tricky to work out what the feed is actually of?

For example, here’s the feed I saw a few days ago:

``` <item>
<title><![CDATA[OU on the BBC: Divine Women  - 9:00pm 25/04/2012 - BBC Two and BBC HD]]></title>
<description><![CDATA[Historian Bettany Hughes reveals the hidden history of women in religion, from dominatrix goddesses to feisty political operators and warrior empresses&nbsp;]]></description>
<location><![CDATA[BBC Two and BBC HD]]></location>
<image title="The Berrill Building">http://www.open.edu/openlearn/files/ole/ole_images/general_images/ou_ats.jpg</image>
<showdate>21:00:00 25/04/2012</showdate>
<pubDate>Tue, 24 Apr 2012 11:19:10 +0000</pubDate>
<dc:creator>sb26296</dc:creator>
</item>```

It contains an upcoming show date for programmes that will be broadcast over the next week or so, and a link to a related page on OpenLearn for the episode, although no direct information about the BBC programme code for each item to be broadcast.

In the meantime, why not see what OU/BBC co-pros are currently available on iPlayer?

Enjoy! :-)

# Tinkering With Scraperwiki – The Bottom Line, OpenCorporates Reconciliation and the Google Viz API

Having got to grips with adding a basic sortable table view to a Scraperwiki view using the Google Chart Tools (Exporting and Displaying Scraperwiki Datasets Using the Google Visualisation API), I thought I’d have a look at wiring in an interactive dashboard control.

You can see the result at BBC Bottom Line programme explorer:

The page loads in the contents of a source Scraperwiki database (so only good for smallish datasets in this version) and pops them into a table. The searchbox is bound to the Synopsis column and and allows you to search for terms or phrases within the Synopsis cells, returning rows for which there is a hit.

Here’s the function that I used to set up the table and search control, bind them together and render them:

```    google.load('visualization', '1.1', {packages:['controls']});

function drawTable() {

var json_data = new google.visualization.DataTable(%(json)s, 0.6);

var json_table = new google.visualization.ChartWrapper({'chartType': 'Table','containerId':'table_div_json','options': {allowHtml: true}});
//i expected this limit on the view to work?
//json_table.setColumns([0,1,2,3,4,5,6,7])

var formatter = new google.visualization.PatternFormat('<a href="http://www.bbc.co.uk/programmes/{0}">{0}</a>');
formatter.format(json_data, [1]); // Apply formatter and set the formatted value of the first column.

formatter.format(json_data, [7,8]);

'controlType': 'StringFilter',
'containerId': 'control1',
'options': {
'filterColumnLabel': 'Synopsis',
'matchType': 'any'
}
});

var dashboard = new google.visualization.Dashboard(document.getElementById('dashboard')).bind(stringFilter, json_table).draw(json_data);

}```

The formatter is used to linkify the two URLs. However, I couldn’t get the table to hide the final column (the OpenCorporates URI) in the displayed table? (Doing something wrong, somewhere…) You can find the full code for the Scraperwiki view here.

Now you may (or may not) be wondering where the OpenCorporates ID came from. The data used to populate the table is scraped from the JSON version of the BBC programme pages for the OU co-produced business programme The Bottom Line (Bottom Line scraper). (I’ve been pondering for sometime whether there is enough content there to try to build something that might usefully support or help promote OUBS/OU business courses or link across to free OU business courses on OpenLearn…) Supplementary content items for each programme identify the name of each contributor and the company they represent in a conventional way. (Their role is also described in what looks to be a conventionally constructed text string, though I didn’t try to extract this explicitly – yet. (I’m guessing the Reuters OpenCalais API would also make light work of that?))

Having got access to the company name, I thought it might be interesting to try to get a corporate identifier back for each one using the OpenCorporates (Google Refine) Reconciliation API (Google Refine reconciliation service documentation).

Here’s a fragment from the scraper showing how to lookup a company name using the OpenCorporates reconciliation API and get the data back:

```ocrecURL='http://opencorporates.com/reconcile?query='+urllib.quote_plus("".join(i for i in record['company'] if ord(i)<128))
try:
except:
recData={'result':[]}
print ocrecURL,[recData]
if len(recData['result'])>0:
if recData['result'][0]['score']>=0.7:
record['ocData']=recData['result'][0]
record['ocID']=recData['result'][0]['uri']
record['ocName']=recData['result'][0]['name']```

The ocrecURL is constructed from the company name, sanitised in a hack fashion. If we get any results back, we check the (relevance) score of the first one. (The results seem to be ordered in descending score order. I didn’t check to see whether this was defined or by convention.) If it seems relevant, we go with it. From a quick skim of company reconciliations, I noticed at least one false positive – Reed – but on the whole it seemed to work fairly well. (If we look up more details about the company from OpenCorporates, and get back the company URL, for example, we might be able to compare the domain with the domain given in the link on the Bottom Line page. A match would suggest quite strongly that we have got the right company…)

As @stuartbrown suggeted in a tweet, a possible next step is to link the name of each guest to a Linked Data identifier for them, for example, using DBPedia (although I wonder – is @opencorporates also minting IDs for company directors?). I also need to find some way of pulling out some proper, detailed subject tags for each episode that could be used to populate a drop down list filter control…

PS for more Google Dashboard controls, check out the Google interactive playground…

# OU Social Media Strategy is a Blast to the Past?!

Readers over a certain age, ex-pats included, will probably remember (hopefully with fondness) a time when the only TV programmes on air in the early hours or on weekend mornings were OU broadcast items on the BBC:

From time to time, (eg OERs: Public Service Education and Open Production), I’ve thought that was the actual heyday of OU broadcasting in terms of get “authentic” Higher Education level teaching content to large audiences, nothwithstanding the popularity of some of the more recent flagship co-produced programming the OU has worked with the BBC on. (For a view of OU/BBC co-produced content currently on iPlayer, see OU/BBC co-pros currently on iPlayer; and for clips from co-pro programmes: clips from OU/BBC co-pros currently on iPlayer.)

As well as the BBC content, there’s also a wealth of OU video material on both YouTube and iTunesU. A great way into this content is through some of the OU’s YouTube playlists, such as 60 Second Adventures in Thought or Seven Wonders of the Microbe World. (See also this full list of OU Learn playlists on YouTube.)

ANyway, one thing that seems (to me at least) to be lacking is a social media strategy (on Twitter at least) relating to broadcast events – academic commentaries or OpenLearn links being tweeted alongside a live OU/BBC co-pro broadcast, for example – that could be used to help drive a second screen experience or community.

But then I realised I was looking in the wrong place – or at least, the wrong time… because it seems the lessons from the past are being heeded… and the @OUpahParr account is actually tweeting out links to OU content to a variety of hashtag streams throughout the early hours, picking up not only the global audience but the UK’s insomniacs and shift workers. It seems that as well as what are presumably scheduled tweets to content, there’s also someone from the comms team (^AF), staffing the account for anybody who wants to chat, or learn more…

Good stuff ;-)

# OU/BBC Co-Pros Currently on iPlayer, via ScraperWiki

A quick update to yesterday’s post on OU/BBC Co-Pros Currently on iPlayer: I’ve popped the first draft of a daily scraper onto Scraperwiki that looks at my delicious bookmark list of OU/BBC series co-pros and tries to find corresponding programmes that are currently available on iPlayer: OU BBC Co-pros on iPlayer Scraperwiki

This is probably not the most efficient solution, but at least it provides some sort of API to at least some relevant iPlayer data.

I’ve also popped up a quick Scraperwiki view over the data OU BBC Co-pros on iPlayer (Scraperwiki HTML View); note that this data is unsorted (I need to think about how best to do that?)

[I’ve added a couple more columns since that screenshot was grabbed; please feel free to work on the scraper, or the view, to improve them further; if you grab a copy of the view to work on your own, please add a link back to it in the comments below, along with a brief description of what you’re trying to achieve with your view…]

PS hmm, maybe I should pop the academics on In Our Time code onto Scraperwiki too?