Hyperlocal Twitter Trends

Just by the by, I idly tweeted last week along the lines of “does anyone know of a twitter trends service that identifies trending topics within a particular region or locale?”.

I didn’t receive any links to such a service at the time (and didn’t build the service myself…) but it strikes me that this could be a really useful hyperlocal news service?

An alternative might be to find ‘trending locations’ or ‘trending places’ rather than trending hashtags or topics, so if there is a sudden flurry of tweets in a particular area, it could get flagged (does Twitter this anyway with its trending topics?).

The locus of the trending regions need not be limited to ‘a circle within fifty miles of a point’ either; they could (with lots of computing power;-) be points along a line, such as a road, for example.

As an asymmetric follower type (I have many more followers than people I follow on Twitter), it might also be handy to be able to see trending topics across the people who follow me, just in case the sampled population that I do follow aren’t a fair sample of the people who follow me…

Just a thought… now back to the jet lag :-(

[See also: Mapping Realtime Events on Twitter and this Simple Embeddable Twitter Map Mashup]

UPDATE: with the release of a Twitter geo api, it didn’t take long: Trendsmap

Twitter Integration Means Delicious Social Bookmarking Site Gets, Err, Social…(?!)

In order to keep track of the dozens of snippets of potentially useful info I come across through m browser each week, I us the delicious online (social) bookmarking sit as a place to dump most of my bookmarks. Three or four times a week (maybe more?), I use my tags to rediscover things I remember bookmarking, and maybe once a month I actually use the delicious search option.

In order to save bookmarks, I used to use the integrated service provided by the flock browser, but in one particular update they made a change I really didn’t get on with (to the dialogue box, I think) and now I tend to use the delicious bookmarklet.

(I’ve also become pretty cavalier about which browser I use – I typically have Flock, Safari and Firefox open, and @dmje was hassling me all last week to use Chrome as well… – so with a bookmarklet on each browser, I get a consistent experience.)

As someone who used to send “FYI” emails out every so often, one of the ways I use twitter is to share potentially interesting or “of the moment” links; I also use a feedthru tag to post one or two links per day, (typically), to my blog sidebar (those links also gt integrated on a daily basis with my blog’s Feedburner feed). Note also that I rarely use the for: option on delicious, possibly because I don’t look at what’s been shared with me very often!

Anyway, one of the hassles with my workflow is the duplicated action required to both tweet and bookmark a link. But it seems that the delicious bookmarklet now has a sharing capability both within and without the delicious ecosystem. (I’m not sure if this is a Good Thing, or a delicious death throe?)

So for example, I can share a bookmark with my delicious network:

social delicious

Or tweet it (it’ll be interesting whether I adopt this workflow…):

So what message gets tweeted? The tweet message, of course:

Note that once the bookmark is saved, there is no evidence or history of it being tweeted. Nor are any of the tags used as hashtags. (If you add a hashtag to the tweet, I don’t think it gets added to the bookmark tags as a simple ‘dehashed’ tag (or hashtag.)

When the link is tweeted, a new delicious shortcode is used:

One Bad Thing about the Twitter integration – you have to provide your twitter credentials. So what the f**k is wrong with OAuth?!

As well as the new social/network amplification options in the bookmark dialogue, there’s also been a revamp (I think) of the search facility:

As well as suggested terms, an improved search display over your own bookmarks, your network’s, or everyone’s, there’s an ability to filter the results by tag. I have to admit I expected live AJAXy UI updates – I didn’t see the effect of filtering by tag unless I clicked the Search button again – but it’s maybe still early days and the live reflow may yet appear. (Or maybe it is there already and just broken for me at the mo?!)

I’m not sure how useful the volume display will be (memories of Google trends etc, there), especially as it only works when you only have one group selected (i.e. only one of my bookmarks, or my network’s, or all of them) and doesn’t reflow when you change the selection? I also wonder how well the ads will fare against the user generated links?

Anyway this is starting to look like it could become quite a powerful search tool, so maybe I need to start growing my delicious network and evangelising once more… (Just a quick note to self – if I do a social bookmarking workshop again, I need to update my slides [uploaded 3 years ago? Sheesh…] ;-)

And as for the twitter integration – I think I’ll give it a go…

PS on the search engine front, I was thinking over the weekend how the mythical ‘social search engine’ that people were trying to hype a year or two ago has actually appeared. But rather than arising out of ‘dead links’ posted to delicious, it’s a live ‘person inside’ application: Twitter.

PPS at last there’s an official announcement post: New and Delicious: Search, Tweet, and Discover the Freshest Bookmarks

Who’s Tweeting Our Hashtag?

Last night, I put togther a quick video showing how to make a Yahoo Pipe that will find who’s been tweeting with a particular hashtag, in particular using the opened09 hashtag:

Here’s the pipe, deconstructed:

The idea behind this pipe in part came from a Dave Winer utility that will create an OPML feed linking to the Twitter feeds from the people that a named Twitter user is following (linked to from rssCloud news); my immediate reaction to that was: would an OPML feed linking to the feeds of everyone who had used a particular hashtag be useful? and this pipe is the result.

(Note to self: I really, really, really need to put a Pipes2OPML script together…)

Anyway, I thought it might make sense to generalise this pipe, and hopefully make it a little more useful, as well as showing a Pipes trick or two in some sort of appropriate context. So here are some problems with the above pipe that I’ll then show you how to fix…

– the pipe is hardwired to search for tweets containing #opened09; how can w generalise it to work with other hashtags/search strings?
– limited number of search results: the pipe only has access to the one hundred most recent opened09 hashtagged tweets; how can we get access to more?
– for a pipe that shows who has been using a particular hashtag, it might also be useful to see how many times they have used that hashtag?
– the pipe displays the hashtag twitterers in the order in which they first tweeted (a history based ordering); which is useful in one sense, but not in another. How else might we order the tweets? Recency of posting? Most active user of the hashtag?

So, first up, how do we generalise the pipe to work with other search terms? Simple, just add a user input for the search term and construct a URI using that term; then wire the URI in to the Fetch Feed block:

A form element appears on the pipes front page, and the search term is also passed via URIs that call the pipe:

Secondly, how do we increase the pool of hashtagged tweets? Simple: Twitter uses paged search results so we can pull in results from those other pages too…

So we can construct URIs for the second, third, fourth etc pages of search results too, and pull feeds in from all those pages:

The third item on our list was displaying how many times each user in the list had used the hashtag (based on the number of times they appeared in the search results sample, of course).

As @ostephens originally pointing out to me (at MashLib09), the Unique block actually counts the number of occurrences of the unique filter term:

We can display this in the title element using a very powerful construction… In the regular expression, rewrite the title string containing the value of the repeatcount variable. How? Like this – access the variable using the construction: ${VARIABLE_NAME}

So in the current example, we can get the value of the count using ${y:repeatcount}:

The final item on our improvements list was to try out some other orderings of the displayed tweets. At the moment, they are ordered according to the date on which each user first used the hashtag term (that is, within the sample of tweets we have pulled back from the Twitter search).

To change the order of names that are output from the pipe, we can use the Sort block. So for example, sort the users in terms of activity:

To sort the users in reverse chronological order (that is, according to who used the hashtag most recently), sort on the published attribute rather than on y:repeatcount.

You can now grab the RSS feed from the pipe and consume it in anything that accepts RSS, pull the JSON feed into your own web page if you’re a little more adventurous, or grab a Google homepage widget from the pipe’s homepage: Hashtag Twitterers Pipe

To tune the pipe to your own needs, you can also clone it from there and then modify your own version of it to your heart’s content:-)

More Thinkses Around Twitter Hashtag Networks: #JISCRI

A brief next step on from Preliminary Thoughts on Visualising the OpenEd09 Twitter Network and A Quick Peek at the IWMW2009 Twitter Network with a couple of graphs that look at the hashtag network around the JISCRI event that’s going on this week.

The sample was a taken from a search of recent #jiscri hashtagged tweets captured last night using the Hashtag Twitterers pipe.

The first chart was to look at people who the hashtag twitterers were following in large numbers who weren’t using the hashtag (I think…my experimental protocol was a bit ropey last night… oops).

The graphs were plotted using Graphviz – firstly a radial plot:

jiscrinetExtGurus

And then a circular one:

jiscrinetExtGurus2

The circular one is quite fun, I think? :-) At a glance, it shows who the “external gurus” are, as well as the differences in their influence.

The second thing I looked at was the network graph of the JISCRI hashtaggers, showing who friended whom:

jiscriTwitterNet

Here’s the circular view:

jiscriTwitterNetCircular

For a large event, I think this sort of graph could be quite fun to generate at both the start of the event and at the end of the event, to show how connections can be formed during an event.

For conferences that publish lists of attendees, popping up a poster of the delegates’ twitter network might provide an interesting discussion thing for people to chat around.

PS See also Meet @HelloApp, Making Conferences More Fun.

Handling Yahoo Pipes Serialised PHP Output

One of the output formats supported by Yahoo Pipes is a PHP style array. In this post, which describes a way of seeing how well connected a particular Twitter user is to other Twitterers who have recently used a particular hashtag, I’ll show you how it can b used.

The following snippet, (cribbed from Coding Forums) shows how to handle this array:

//Declare the required pipe, specifying the php output
$req = "http://pipes.yahoo.com/ouseful/hashtagtwitterers?_render=php&min=3&n=100&q=%23jiscri";

// Make the request
$phpserialized = file_get_contents($req);

// Parse the serialized response
$phparray = unserialize($phpserialized);

//Here's the raw contents of the array
print_r($phparray);

//Here's how to parse it
foreach ($phparray['value']['items'] AS $key => $val)
	printf("<div><p><a href=\"%s\">%s</a></p><p>%s</p>\n", $val['link'], $val['title'], $val['description']);

The pipe used in the above snippet (http://pipes.yahoo.com/ouseful/hashtagtwitterers) displays a list of people who have recently used a particular hashtag on Twitter a minimum specified number of times.

It’s easy enough to parse out the Twitter ID of each individual, and then for a particular named individual see which of those hashtagging Twitterers they are either following, or are following them. (Why’s this interesting? Well, for any given hashtag community, it can show you how well connected you are with that community).

So let’s see how to do it. First, parse out the Twitter ID:

foreach ($phparray['value']['items'] AS $key => $val) {
	$id=preg_replace("/@([^\s]*)\s.*/", "$1", $val['title']);
	$idList[] = $id; 
}

We have the Twitter screennames, but now we want the actual Twitter user IDs. There are several PHP libraries for accessing the Twitter API. The following relies on an old, rejigged version of the library available from http://github.com/jdp/twitterlibphp/tree/master/twitter.lib.php (the code may need tweaking to work with the current version…), and is really kludged together… (Note to self – tidy this up on day!)

The algorithm is basically as follows, and generates a GraphViz .dot file that will plot the connections a particular user has with the members of a particular hashtagging community:

  • get the list of hashtagger Twitter usernames (as above);
  • for each username, call the Twitter API to get the corresponding Twitter ID, and print out a label that maps each ID to a username;
  • for the user we want to investigate, pull down the list of people who follow them from the Twitter API; for each follower, if the follower is in the hashtaggers set, print out that relationship;
  • for the user we want to investigate, pull down the list of people who they follow (i.e. their ‘friends’) from the Twitter API; for each friend, if the friend is in the hashtaggers set, print out that relationship;
$Twitter = new Twitter($myTwitterID, $myTwitterPwd);

//Get the Twitter ID for each user identified by the hashtagger pipe
foreach ($idList as $user) {
	$user_det=$Twitter->showUser($user, 'xml');
 	$p = xml_parser_create();
	xml_parse_into_struct($p,$user_det,$results,$index);
	xml_parser_free($p);
	$id=$results[$index['ID'][0]][value];
	$userID[$user]=$id;
	//print out labels in the Graphviz .dot format
	echo $id."[label=\"".$user."\"];\r";
}

//$userfocus is the Twitter screenname of the person we want to examine
$currUser=$userID[$userfocus];
 
//So who in the hashtagger list is following them?
$follower_det=$Twitter->getFollowers($userfocus, 'xml');
$p = xml_parser_create();
xml_parse_into_struct($p,$follower_det,$results,$index);
xml_parser_free($p);
foreach ($index['ID'] as $item){
	$follower=$results[$item][value];
	//print out edges in the Graphviz .dot format
	if (in_array($follower,$userID)) echo $follower."->".$currUser.";\r";
}

//And who in the hashtagger list are they following?
$friends_det=$Twitter->getFriends($userfocus, 'xml');
$p = xml_parser_create();
xml_parse_into_struct($p,$friends_det,$results,$index);
xml_parser_free($p);
foreach ($index['ID'] as $item){
	$followed=$results[$item][value];
	//print out edges in the Graphviz .dot format
	if (in_array($followed,$userID)) echo $currUser."->".$followed.";\r";
}

For completeness, here are the Twitter object methods and their associated Twitter API calls that were used in the above code:

function showUser($id,$format){
	$api_call=sprintf("http://twitter.com/users/show/%s.%s",$id,$format);
  	return $this->APICall($api_call, false);
}

function getFollowers($id,$format){
  	$api_call=sprintf("http://twitter.com/followers/ids/%s.%s",$id,$format);
 	return $this->APICall($api_call, false);
}
  
function getFriends($id,$format){
  	$api_call=sprintf("http://twitter.com/friends/ids/%s.%s",$id,$format);
 	return $this->APICall($api_call, false);
}

Running the code uses N+2 Twitter API calls, where N is the number of different users identified by the hashtagger pipe.

The output of the script is almost a minimal Graphviz .dot file. All that’s missing is the wrapper, e.g. something like: digraph twitterNet { … }. Here’s what a valid file looks like:

(The labels can appear either before or after the edges – it makes no difference as far as GraphViz is concernd.)

Plotting the graph will show you who the individual of interest is connected to, and how, in the particular hashtag community.

So for example, in the recent #ukoer community, here’s how LornaMCampbell is connected. First a ‘circular’ view:

ukoerInternalNetLMC2

The arrow direction goes FROM one person TO a person they are following. In the circular diagram, it can be quite hard to see whether a connection is reciprocated or one way.

The Graphviz network diagram uses a separate edge for each connection and makes it easier to spot reciprocated links:

ukoerInternalNetLMC

So, there we have it. Another way of looking at Twitter hashtag networks to go along with Preliminary Thoughts on Visualising the OpenEd09 Twitter Network, A Quick Peek at the IWMW2009 Twitter Network and More Thinkses Around Twitter Hashtag Networks: #JISCRI

Personal Twitter Networks in Hashtag Communities

Another conference I’m not at, this time ALT-C, so time for another blatant attempt to raise my profile at the event even if I’m not there with another Twitter related hack…;-) This time, a little tool to help you explore the extent of your Twitter network within a community of people using a particular hashtag.

Here’s a tease of the sort of report it gives:

My place in a Twitter hashtag community http:ouseful.open.ac.uk/twitterMyhashtagNet.php?q=psychemedia&h=altc2009

Some numbers (I’ll let you know what in a minute…) A list of people in the hashtag network who are followed by a particular individual (their “friends”). A list of people in the hashtag network who follow a particular individual, but are not followed (friended) back (their “serfs”). A list of people in the hashtag network who are followed (friended) a particular individual, but do not follow them (their “slebs”). A list of people in the hashtag network who neither follow nor are followed (friended) by a particular individual (“the void”).

Before I go on, I should probably also define what I mean by a hashtag community, not last because there are some, err, pragmatic constraints on defining this;-)

For the purposes of this post, a hashtag community is a collection of people who have used a particular hashtag more than a certain minimum specified number of times in a set of Twitter posts that use the hashtag. In my default ad hoc set up, I tend to look for people who have used the hashtag more than 3 times in the most recent 500 or so tweets. For the proof of concept demo, I also limit the size of the hashtag network to 100, otherwise the pipework that underpins it starts to fall over…

UPDATE:
Here’s a bit more explanation about why the app doesn’t always show people in the community you ‘know’ to be there…

You may notice that not everyone you know has used the hashtag appears in the friends and followers lists. This is because the size of the hashtag community is limited in three ways:

  • hashtag use sample size: for this proof of concept, the hashtag community analysis is based on a Twitter search that grabs the 500 most recent uses of the declard hashtag. If this were a production tool, it would pull the complete archive of hashtag use from one of the twitter archiving services. if you want that feature, build it yourself…;-)
  • minimum number of tweets: an optional paramenter in the URI identifies the minimum number of hashtagged tweets that a user must have sent in the sample to be considered a member of the community. By setting this numbr large, it allows you to just see the heaviest hashtagger in the community, or filter out people who maybe just use the hashtag once in a retweet. (I think there’s a bug in the code – if you set this mintweets paramter to 2, the user must have hashtagged at least 3 times. i.e. one more. 10 is 11.
  • Max community size: an ‘issue’ in the Twitter search API means I need to call the Twitter API once for every person in the community. This overhead can break the pipework, so the community size can be limited arbitrarily.

The inspiration for the report is a typical ego thing – to what extent is my personal Twitter network dominated by the membership of a particular hashtag community. (Note I’ve explored related ideas in a variety of other ad hoc ways: Who’s Tweeting Our Hashtag?, Where Next With The Hashtagging Twitterers List?, Preliminary Thoughts on Visualising the OpenEd09 Twitter Network, A Quick Peek at the IWMW2009 Twitter Network, More Thinkses Around Twitter Hashtag Networks: #JISCRI and Handling Yahoo Pipes Serialised PHP Output).

Anyway, in the current example, the numbers I’ve started to look at are defined as follows. All numbers are either integers, or real numbers in the range 0..1.

So what do the numbers mean?

  • Number of hashtaggers: the number of people in the hashtag network, Ngalaxy;
  • Hashtaggers as followers (‘hashtag followers’): the number of people in the hashtag community who are following the named individual, Gfollowers
  • Hashtaggers as friends (‘hashtag friends’): the number of people in th hashtag community that the named individual has friended, Griends
  • Hashtagger followers not friended (‘serfs’): the number of people in the hashtag community that follow the named individual but that are not followed back (i.e. who are not friends of the named individual), Gserfs
  • Hashtagger friends not following (‘slebs’): the number of people in the hashtag community that are followed by the named individual (i.e. friends) but that do not follow them back (i.e. who are not also followers of the named individual), Gslebs
  • Hashtaggers not friends or followers (‘the hashtag void’): the number of people in the hashtag community who neither follow, nor are friended by, the named individual Gvoid
  • Reach into hashtag community: the proportion of the the hashtag community that follow the named individual; a measure of the extent to which an individual can reach the hashtag community without actually using the hashtag; Greach=Gfollowers/Ngalaxy.
  • Reception of hashtag community the proportion of the the hashtag community that are followed by (i.e. are friends of) the named individual; a measure of the extent to which an individual sees messages from the hashtag community without directly tracking the hashtag; Greception=Gfriends/Ngalaxy
  • Hashtag void (normalised): the size of the void normalised relative to the size of the hashtag community; the proportion of the hashtag community that are unlikely to be directly encountered outside of the hashtag community; Normvoid=Gvoid/Ngalaxy
  • Total personal followers the total number of followers of the named individual, Nfollowers
  • Total personal friends: the total number of friends of the named individual Nfriends
  • Hashtag community dominance of personal reach: the extent to which the hashtag community dominates the set of people who follow the named individual, Domreach=Gfollowers/Nfollowers. If all the named individual’s followers are in the hashtag community, Domreach=1. If none of them are, Domreach=0.
  • Hashtag community dominance of personal reception: the extent to which the set of the named individual’s friends is dominated by members of the hashtag community, Domreception=Gfriends/Nfriends. If all the named individual’s friends are in the hashtag community, Domreception=1. If none of them are, Domreception=0.

If you want to try the tool out, the interface is provided by the URI:
http://ouseful.open.ac.uk/twitterMyhashtagNet.php?q=ostephens&h=altc2009&mintweets=2&maxusers=99

I have no idea whether any of these measures are used in more formal analyses (I’ve yet to start my formal reading of the proper social network analysis stuff…) but it’s a way in for me to start thinking about what measures that might be in some sense meaningful and both easy to explain and calculate;-)

Searching for Twitter Hashtags and Finding Hashtag Communities

Over the last few weeks I’ve been messing around more than I should with Twitter, and in particular trying to get a feel for how we might use hashtag communities as a well of identifying and growing community structures in a particular topic area (see posts all over OUseful.info for more details).

A couple of days ago, @clarileia raised the question of how you find new hashtags, so I had a little tinker today putting together a couple of hacks (Twitter hashtag search pipe and Twitter my network hashtags) that let you identify recently used Twitter hashtags associated with a particular search term, or with a specified user’s recent friends or followers.

Twiitter hashtag search http://pipes.yahoo.com/ouseful/twitterhashtagsearch

[Note: at the time of writing, Pipes appears to be running a little slow… if the Pipe appears to stall, it does work, honest… try it again later ;-)]

At the core of all the hacks is a clunky hashtag tokeniser pipe that takes a Twitter status update and pulls out the hashtags:

This utility pipe works by taking the status update, extracting the hashtags using a hacked together regular expression, splits the separate hashtags into separate feed items, and then filters them to emit only legitimate hashtags.

The utility pipe is then used in a search powered pipe, which searches twitter for the 100 most recent tweets containing the search terms and then scans those for hashtags; and a ‘personal network hashtags’ pipe that takes a Twitter username, pulls back the tweets from their one hundred most recent friends, and their one hundred most recently followers, and then scans those tweets for hashtags.

For example, here’s the search pipe:

Both pipes have a common output routine – the list of hashtags is filtered through the Unique block, which also returns a count of how many times each hashtag has appeared. The hashtags are then ordered and filtered according to the minimum number of required occurrences in the sample. A regular expression adds the number of occurrences of each hashtag.

The pipes could be extended to pull in more search results, or more followers/friends (maybe the first hundred friends/followers as well as the most recent hundred?) but that’s left as an exercise for the reader. As for the use case – I dunno? Maybe integration with the OUseful TwitterMyHashtag apps? Or perhaps @clarileia had a use case in mind?!;-)

PS thanks to PJ on the Yahoo Pipes team for getting back to me earlier today when I was struggling with a slow running pipes editor… I’m now totally reliant on Pipes for many apps, and especially for rapid the majority of my prototyping, so when Pipes is slow, I feel as happy as if I’ve lost an unbacked up server… Brian Kelly would probably tell me I need to do a risk assessment… I’ve already done one: What Happens If Yahoo! Pipes Dies? – but I haven’t made a start on the contingency stuff that was considered there…