eduTwitterin’

Jane’s list of “100+ (E-)Learning Professionals to follow on Twitter” (which includes yours truly, Martin and Grainne from the OpenU :-) has been doing the rounds today, so in partial response to Tony Karrer asking “is there an equivalent to OPML import for twitter for those of us who don’t want to go through the list and add people one at a time?”, I took an alternative route to achieving a similar effect (tracking those 100+ e-learning professionals’ tweets) and put together a Yahoo pipe to produce an aggregated feed – Jane’s edutwitterers pipe

Scrape the page and create a semblance of a feed of the edutwitterers:

Tidy the feed up a bit and make sure we only include items that link to valid twitter RSS feed URLs (note that the title could do with a little more tidying up…) – the regular expression for the link creates the feed URL for each edutwitterer:

Replace each item in the edutwitterers feed with the tweets from that person:

From the pipe, subscribe to the aggregated edutwitters’ feed.

Note, however, that the aggregated feed is a bit slow – it takes time to pull out tweets for each edutwitterer, and there is the potential for feeds being cached all over the place (by Yahoo pipes, by your browser, or whatever you happen to view the pipes output feed etc. etc.)

A more efficient route might be to produce an OPML feed containing links to each edutwitterer’s RSS feed, and then view this as a stream in a Grazr widget.

Creating the OPML file is left as an exercise for the reader (!) – if you do create one, please post a link as a comment or trackback… ;-) Here are three ways I can think of for creating such a file:

  1. add the feed URL for each edutwitter as a separate feed in an Grazr reading list (How to create a Grazr (OPML) reading list). If you don’t like/trust Grazr, try OPML Manager;
  2. build a screenscraper to scrape the usernames and then create an output OPML file automatically;
  3. view source of Jane’s orginal edutwitterers page, cut out the table that lists the edutwitterers, paste the text into a text editor and work some regular ecpression ‘search and replace’ magic; (if you do this, how about posting your recipe/regular expressions somewhere?!;-)

Enough – time to start reading Presentation Zen

Simple Embeddable Twitter Map Mashup

Earlier today, I was pondering the Digital Planet Listeners’ map on open2.net and the #digitalplanet Twitter feed we have running alongside it:

and I started to wonder whether there was a simple way of generating an embeddable map showing the location of people tweeting given a Twitter feed.

I couldn’t find one offhand, so here’s a simple pipe that will do the job: Simple Tweetmap pipe.

Here’s how it works: start off by grabbing a feed from Twitter, such as a Twitter search feed.

Using a Twitter feed URL as an input to the pipe, grab the feed and then find the Twitter username of each individual from the user’s Twitter URL. So for example, map http://twitter.com/psychemedia onto psychemedia.

We now call on another pipe that calls the Twitter API to get personal details for each user who has a Tweet in the feed.

Here’s how that embedded pipe works: Twitter location pipe (it should really be called “Twitter User Details” pipe).

First, construct a URI that ask the Twitter API for the user details associated a particular Twitter username, (e.g. using the construction http://twitter.com/users/show/USERNAME.json) then pull the data back as a JSON feed. Finally, just make sure only a single record is returned (there should only be one anyway).

So the embedded pipe passes back an annotation to the original feed with user details. One of the user details is the user’s location – so let’s geocode it:

Sending the output of the location finder to item.y:location allows the pipe to produce a well formed KML and geoRSS output, that can be displayed in a map, such as the Yahoo! Pipes output preview map:

We can grab the KML URL from the More Options output and display the feed in an embeddable Google map in the normal way (simply enter the KML URI in the Google maps search box and hit Search):

If you want to embed the map in your own page, just grab the embed code…

To summarise the hack, here’s a quick review of the whole pipe:

So now if you want to plot where people who have tagged their tweets in a particular way are tweeting from, you can do :-)

Next step is to persuade the open2 team to start archiving appropriately tagged tweets and displaying them on a cluster map view over time :-) We could maybe even link in a timeline using something like TimeMap, the MIT Simile timeline and Google Maps integration library…?

Twitter Powered Subtitles for Conference Audio/Videos on Youtube

Last week, I wrote a post on hHow people Tweeted Through Carter on “Delivering Digital Britain” at NESTA, where I created a slideshow of tweets posted in response to the NESTA Delivering Britain event/video stream and used them to illustrate the audio recording of Lord Carter’s presentation.

Chatting to @liamgh last week, I mentioned how i was stumped for an easy way to do this. He suggested creating a subtitles feed, and then uploading it to Youtube, along with the audio recording (doh!).

So – here’s a proof of concept result of doing just that:

(Actually, I’m not sure that subtitles/closed captions work in the embedded movies, so you may have to click through: proof of concept use of Tweets as subtitles: Carter at ‘Delivering Digital Britain’, part 1. NB it is possible force embedded videos to show captions, as
Learn More: Showing captions by default
describes. Simply add &cc_load_policy=1 to the embed code; so e.g. in WordPress, you would use something like [youtube=http://www.youtube.com/watch?v=tBmFzF8szpo&cc_load_policy=1 ].
)

And here’s how I did it…

The first thing to do was check out how Youtube handled subtitles: Getting Started: Adding / Editing captions.

The trick is to upload a textfile with lines that look something like this:
0:03:14.159
Text shown at 3 min 14.159 sec for an undefined length of time.

0:02:20.250,0:02:23.8
Text shown at 2 min 20.25 sec, until 2 min 23.8 sec

Secondly – getting the list of tweets hashtagged with #carter over the period Lord Carter was speaking (i.e. the period covered by the video). For the original proof of concept, I used the tweets from the spreadsheet of scraped tweets that @benosteen grabbed for me, though it later occurred to me I could get the tweets direct from a Twitter search feed (as I’ll show in a minute).

Scraped tweet timestamp

The question now was how to get the timecode required for the subtitles file from the timestamp associated with each tweet. Note here that the timecode is the elapsed time from the start of the video. The solution I came up with was to convert the timestamps to universal time (i.e. seconds since midnight on January 1st 1970) and then find the universal time equivalent of the first tweet subtitle; subtracting this time from the universal time of all the other tweets would give the number of seconds elapsed from the first tweet, which I could convert to the timecode format.

As to how I got the universal time values – I used a Yahoo pipe (Twitter CSV to timeingsd):

At this point, it’s probably worth pointing out that I didn’t actually need to call on @benosteen’s tweetscraper – I could just use the Twitter search API (i.e. the Twitter advanced search feed output) to grab the tweets. How so? Like this:

Looking at the results of this query, we see the timing is a little off – we actually need results from 8.30am, the actual time of the event:

Which is where this comes into play – searching for “older” results:

If you click on “Older” you’ll notice a new argument is introduced into the search results page URL – &page=:

…which means that by selecting appropriate values for rpp= and page= we can tunnel in on the results covering from a particular time by looking at “older” results pages, and grabbing the URL for the page of results covering the time period we want:

Then we can grab the feed:

which gives us something like this:
http://search.twitter.com/search.atom?q=+%23carter+since%3A2009-02-24+until%3A2009-02-24

which we hack a little to get to the right period:

http://search.twitter.com/search.atom?q=+%23carter+since%3A2009-02-24+until%3A2009-02-24&page=4&rpp=100

(For more info on the Twitter search API, see the Twitter Search API Documentation wiki.)

NB while I’m at it, note that there’s a corollary hack here that might
come in useful somewhere, or somewhen, else – getting a Twitter search feed into a Google spreadsheet (so we can, for example,process it as a CSV file published from the spreadsheet):

=ImportFeed("http://search.twitter.com/search.atom?rpp=100&page=4&q=+%23carter+since%3A2009-02-24+until%3A2009-02-24")

That is:

Okay – back to the main thread – and a tweak to the pipe to let us ingest the feed, rather than the spreadsheet CSV:

Just by the by, we can add a search front end to the pipe if we want:

and construct the Twitter search API URI accordingly:

(The date formatter converts the search date to the format required by the Twitter search API; it was constructed according to PHP: strftime principles.)

Ok – so let’s recap where we’re at – we’ve now got a pipe that will give us universal timecoded tweets (that’s not so far for such a long post to here, is it?!) If we take the JSON feed from the pipe into an HMTL page, we can write a little handler that will produce the subtitle file from it:

Here’s the code to grab the pipe’s JSON output into an HTML file:

var pipeUrl="http://pipes.yahoo.com/pipes/pipe.run?_id=Dq_DpygL3hGV7mFEAVYZ7A&aqs";

function ousefulLoadPipe(url){

var d=document;
var s=d.createElement('script');
s.type='text/javascript';

var pipeJSON=url+"&_render=json&_callback=parseJSON";
s.src=pipeJSON;
d.body.appendChild(s);
}
ousefulLoadPipe(pipeUrl);

Here’s the JSON handler:

function parseJSON(json_data){
var caption; var timestamp=0; var mintime=json_data.value.items[0]['datebuilder'].utime;
for (var i=0; itimestamp) mintime=timestamp;
}

for (var j=json_data.value.items.length-1; j>=0; j--) {
caption=json_data.value.items[j]['title'];
timestamp=1*json_data.value.items[j]['datebuilder'].utime;
if (j>0) timeEnd=(1*json_data.value.items[j-1]['datebuilder'].utime)-3; else timeEnd=10+1*json_data.value.items[j]['datebuilder'].utime;
if (timeEnd<timestamp) timeEnd=timestamp+2;
timecode=getTimeCode(timestamp-mintime);
timeEnd=getTimeCode(timeEnd-mintime);
var subtitle=timecode+","+timeEnd+"
"+caption+"<br/><br/>";
document.write(subtitle);
}
}

Here’s the timecode formatter:

//String formatter from: http://forums.devshed.com/javascript-development-115/convert-seconds-to-minutes-seconds-386816.html
String.prototype.pad = function(l, s){
return (l -= this.length) > 0 ? (s = new Array(Math.ceil(l / s.length) + 1).join(s)).substr(0, s.length) + this + s.substr(0, l - s.length) : this; };

function getTimeCode(seconds){
var timecode="00:"+(Math.floor(seconds / 60)).toFixed().pad(2, "0") + ":" + (seconds % 60).toFixed().pad(2, "0")+".0";
return timecode;
}

(I generated the timecodes in part using a string formatter from http://forums.devshed.com/javascript-development-115/convert-seconds-to-minutes-seconds-386816.html.)

Copy and paste the output into a text file and save it with the .sub suffix, to give a file which can then be uploaded to Youtube.

So that’s the subtitle file – how about getting the audio into Youtube? I’d already grabbed an audio recording of Carter’s presentation using Audacity (wiring the “headphones out” to the “microphone in” on my laptop and playing the recording from the NESTA site), so I just clipped the first 10 minutes (I think Youtube limits videos to 10 mins?) and saved the file as a wav file, then imported it into iMovie (thinking I might want to add some images, e.g. from photos of the event on flickr). This crib – iMovie Settings for Upload to YouTube – gave me the settings I needed to export the audio/video from my old copy of iMovie to a file format I could upload to Youtube (I think more recent versions of iMovie support a “Share to Youtube” option?).

I then uploaded this file, along with the subtitles file:

So there we have it: Twitter subtitle/annotations (pulled from a Twitter search feed) to the first part of Lord Carter’s presentation at Delivering Digital Britain…

PS Also on the Twitter front, O’Reilly have started watching Twitter for links to interesting stories, or into particular debates: Twitscan: The Debate over “Open Core”.

Chatting to @cheslincoln the other night, we got into a discussion about whether or not Twitter could be used to support a meaningful discussion or conversation, given the immediacy/short lived nature of tweets and the limited character count. I argued that by linking out to posts to support claims in tweets, “hyper-discussions” were possible. By mining “attention trends” (a term I got from misreading a tweet of Paul Walk’s that scaffold a conversation, it’s possible to create a summary post of a conversation, or argument, like the O’Reilly one?

See also this post from Paul Walk: Anything you quote from Twitter is always out of context.

Twitter Powered Youtube Subtitles, Reprise: Anytime Commenting

One of the things that attracts me to serialised feeds (as well as confusing the hell out of me) is the possibility of letting people subscribe to, and add, comments in “relative time”…

… that is, as well as viewing the content via a serialised feed, the comments feed should also be serialised (with timestamps for each comment calculated relative to the time at which the person commenting started receiving the serialised feed).

Applying this to the idea of tweeted Youtube movie subtitles (Twitter Powered Subtitles for Conference Audio/Videos on Youtube) in which every tweet made during a presentation at or around that presentation becomes a subtitle on a recording of that presentation, it strikes me that a similar model is possible.

That is, different individuals could watch a Youtube video at different times, tweeting along as they do so, and then these tweets could be aggregated according to relative timestamps to provide a single, combined set of subtitles.

So how might this work in practice? Here’s a thought experiment run through…

Firstly, it’d probably be convenient to set up a twitter account to send the tweets to (say @example, for example).

Create a tag for the video – this could be something like #yt:tBmFzF8szpo for the video at http://www.youtube.com/watch?v=tBmFzF8szpo.

(Alan Levine reminded me about flickr machine tags earlier today, which are maybe also worth considering in this respect, e.g. as a source of inspiration for a tagging convention?)

Grab a ctrl-C copy of the phrase @example #yt:tBmFzF8szpo for quick pasting into a new tweet, and then start watching the video, tweeting along as you do so…

To generate your subtitle feed, you can then do a search based on Tweets from your username (which would be @psychemedia in my case) to e.g. @example, with hashtag #yt:tBmFzF8szpo, and maybe also using a date range.

(You could augment the Yahoo pipe I used in the twitter subtitle generator proof of concept to remove the hashtag when generating the feed used for subtitling?)

The actual subtitle file generator could then pull in several different subtitle feeds from separate people, relativise their timestamps relative to the time of the first tweet (which could maybe use a keyword, too – such as “START”: @example START #yt:tBmFzF8szpo;-) and then produce an aggregated subtitle feed.

As more people watched the video (maybe including the subtitles to date), their feeds could be added to the aggregating subtitle file generator, and the subtitle file updated/refreshed.

Individuals could even rewatch the video and create new feeds for themselves to join in the emerging conversation…

(Okay, so it’s maybe slower than just reading through the comments, having to replay the video in real time to read the tweets, but this is a sort of thought experiment, right, albeit one that can be implemented quite easily…;-)

PS In one of the comments to Show and Translate YouTube Captions Matt Cutts gave an example of a URL that “will search for the word “china” in videos with closed captions” [ http://www.youtube.com/results?closed_captions=1&search_query=china ] (although I’m not sure how well it works?).

So I’m thinking – if live tweets from an event can be associated with a video of an event (maybe because the video is posted with a link to a (now out of date!) upcoming record for that event in order to anchor it in time) then being able to search the tweets as captions/subtitles provides a crib for deeplink searching into the video? (But then, I guess the Goog is looking at audio indexing anyway?)

PPS I just came across another tool for adding subtitles to Youtube videos, as well as videos from other online video sites – overstream.net:

It’s worth looking at, maybe?

PPPS see also Omnisio, a recent Google acquisition that offers “select clips from videos you find on YouTube and other video sites, and easily post them on your profile page or blog. Even better, you and your friends can add comments directly in the video!”.

And there’s more: “With Omnisio you make and share your own shows by assembling clips from different videos.” Roll on the remixes :-)

PPPPS Martin implemented anytime commenting

Anti-tags and Quick and Easy Block (Un)commenting

Looking back over the comments to @benosteen‘s post on Tracking conferences (at Dev8D) with python, twitter and tags just now, I noticed this comment from him replying to a comment from @miaridge about “app noise” appearing in the hashtag feed:

@mia one thing I was considering was an anti-tag – e.g. #!dev8d – so that searches for ‘dev8d’ would hit it, but ‘#dev8d’ shouldn’t.

The other tweak to mention is that booleans work on the twitter search:

‘#dev8d -from:randomdev8d’ would get all the #dev8d posts, but exclude those from randomdev8d.

Likewise, to get all the replies to a person, you can search for ‘to:username’, handy to track people responding to a person.

Brilliant:-)

Note also that one thing worth bearing in mind when searching on Twitter is that a search for @psychemedia is NOT the same as a search for to:psychemedia. That is, those two searches may well turn up different results.

The “to:” only searches for tweets that START with “@pscyhemedia”; so id @psychemedia appears elsewhere in the tweet (e.g. “waht the ??? is @psychemedia talking about?”), the “to:” search will not find it, whereas the “@” search will.

Why’s this important? Well, a lot of people new to using Twitter use the Twitter website interface to manage their interactions, the the “Replies” screen is powered like the “to:” search. Which means if someone “replies” to you in a “multiple addressee” tweet – e.g. “@mweller @psychemedia are you gonna make some more edupunk movies?”, then if you’re not the first named person, the @Replies listing won’t show the tweet… the only way you can discover them is to search twitter for “@psychemedia”, for example.

The Twitter advanced search option to search for posts “Referencing a person” is simply a search of the @person form.

(Note that Twitter search lets you subscribe to search results – so you can always subscribe to an ego search feed and receive updates that way; or you can use a client such as Tweetdeck which runs the search automatically.)

(I’m not sure what happens if someone actually replies to one of your tweets and then prepends some text before your name? Will twitter still spot this as a reply? If anyone knows, can you please comment back?)

Just by the by, the “anti-tag” trick reminds me of this code commenting pattern/trick (I don’t remember where I came across it offhand?) that makes it easy to comment and uncomment blocks of code (e.g. in PHP or Javascript):

Before the comment…]
/*
This will be commented out…
//*/
After the comment…

To uncomment out the block of code, just add a leading “/” to the first comment marker to turn it into a single line comment:

Before the comment…]
//*
This will NO LONGER be commented out…
//*/
After the comment…

The block comment closing tag is now itself single line commented out.

(I seem to remember it was described far more eloquently than that when I came across it!;-)

PS Ah ha, here we are – where I first saw the commenting trick: Every keystroke is a prisoner – a neat commenting trick.

Mapping Realtime Events on Twitter

One of the nice things about blogging in WordPress is the dashboard report that shows which other blog pots are linking in to your blog. One of my posts that’s had a couple of incoming links lately is Simple Embeddable Twitter Map Mashup, firstly from this post on TweetMapping Archaeology and then from Twitter Watermain Mapping – Part Two.

This latter post – plotting out tweets around the occurrence of breaks in watermains – also plots out a map showing people twittering about stormwater.

Which got me thinking, how about a Twittermap to plot tweets about electricity, gas or water being cut off?

By altering the search term, you can search for other events, such as earthquake or bee swarm:

If you want to search around a particular location, then this pipe may be more useful- locale based twittermap (the default search is for the word bid, but it works equally well if you’re wondering where the fire is):

Finally if you’d rather just use the URL for a Twitter search feed as the basis for a map, this pipe should do: map a Twitter search feed URL.

Brand Association and Your Twitter Followers

One of the thing’s that Martin appears to have been thinking a lot about lately are metrics for rating ‘digital scholars’, i.e. those of us who don’t do any of the reputation bearing thing that traditional academics do, (though whether that’s because we’re not very good at those things is not for me to say;-)

So for example, in The Keynote Equivalent? he reviews the notions of reputation and impact factor, and in Connectors versus outputs he calls on some really dubious social media buzz metrics to raise the more far more valid issue of how we measure influence within, and value the contributions made to, a social network (peer community?) in order to recognise the extent of someone’s influence within that network from outside of it.

Using Twitter as a base case, one of the many interesting features of Twitter’s ‘open privacy’ model is that in most cases it’s possible for you to look at someone else’s followers to see who they are.

The value of that network to an individual is at least twofold – firstly, as a source of information, observations, news and feedback to you as the person at the centre of your own network; secondly as an amplifier of your own ego broadcast messages. (There are other benefits of course – like being able to see who is talking to whom about what.) You may also feel there is some benefit to just having a large number of followers, if only in the bragging stakes.

That is, the more followers the better, right? It’s bound to be good for my reputation, if nothing else, surely…?

Well….. maybe not…?

Consider these two questions:

who follows you? if I look at your followers what can I tell about you, from them?
what is your blocking policy? who you block is just as much a part of the way you manage your network as the people you actively follow.

As far as my own Twitter network goes, I am on a follow:followed ratio of about 1:4. That is, approximately four times as many people follow me as I follow back. For every 10 or so new followers I get, I block one or two.

I check my followers list maybe once every two or three days, which lets me keep up with the pruning on just one or two screens of followers using the Twitter web interface. If the name or avatar is suspect, I’ll check out the tweets to see if I want to block. (I really miss the ability to hover over a person’s name and get a tooltip containing their bio:-( If the name or avatar is familiar or intriguing, I’ll check the tweets to see if I’m going to follow back (maybe 1 in 20? Following back is not the main source for me of new people to follow – you’ll have to get to me another way;-).

The people I block? People who’s tweets are never replies, but who just tweet out advertising links all the time; Britney, whatever she happens to be sucking or loving at the time; product tweeters; and so on. If you’re following lots of people and only followed by a few? Not good – why should I follow you if no-one else does? If you’re following lots of people and are followed by lots of people? Also not good: either you’re a spammer being spammed back, or you’re an indiscriminate symmetric follower so why should I trust you, or you’ve so many followers I’m not going to get a look in. If I’m not sure about a new follower, it’s 50/50 that I’ll either block them or not, so there may well be the odd false positive amongst the people I’ve blocked (if so, sorry…) And why do I block them? Because they add no value to me… Like junk mail… And because by association, if you look at my followers and see they’re all Britney, you’ll know my amplification network is worthless. And by association… ;-)

The people I follow? People I’ve chatted to, have been introduced to through RTs, or via interesting/valuable multiaddressed tweets that include me; people who appear not to be part of any other network I follow (or who might add value in a sphere of influence or interest that I don’t feel I currently benefit from), and so on.

And the people I don’t follow but don’t block (i.e. the majority) – nothing personal, but I only have so many hours in the day, and can’t cope with too many new messages every update cycle in by twitter client!

So all this might sound a little bit arrogant, but it’s my space and it’s me that has to navigate it!

PS just by the by, it struck me during an exchange last week that networks can also act as PR channels. A tweet went out from @ruskin147 asking if anyone knew anyone “who can analyse how viral emails,campaigns etc, can knock a firm off course?” Now I should probably have recommended someone from the OU Business School, because I think there is someone there who knows this stuff; but they’re not part of any of my networks so I’d have to go and search for them and essentially recommend them cold. So instead I suggested @mediaczar (who blogs under the same ID) because he’s been sharing code and insight about his analysis of connectivity and the flow of ideas across social networks for the PR firm (I think?) he works for. (Some irony there, methinks?;-) And it turned out that the two of them hooked up and had a chat…

So why’s that good for me? Because it strengthened the network that I inhabit. It increased the likelihood of those people having an interesting conversation that I was likely to also be interested in. I get value not just from people telling me things, but also from people in my network telling each other things that I am likely to find interesting.

And as a spin-off, it maybe increases my reputation with those two people for having helped create that conversation between them?

In terms of externally recognised value though? How are you going to measure that, Martin?

See also: Time to Get Scared, People?