OUseful.Info, the blog…

So it seems that my to-do list just got shorter as Twitter acquire BackType and as a result “will discontinue the BackType product and API services”.

Bah…:-(

On my roadmap (err, such as it is!;-), one thing I was hoping to do was start exploring in more detail the struture of communities around a shared link, with a view to exploring in more detail some of the actual dynamics of link sharing across Twitter networks. My early forays in to this have tended to use BackType, as for example in Visualising Ad Hoc Tweeted Link Communities, via BackType.

The simple recipe I’d started out with was based around the following steps:

- given the URL, look up who’s tweeted it via the BackType API;
- for each tweeter of the link, grab the list of people they follow (i.e. their friends);
- plot the “inner” network showing which of the people who tweeted the link the follow each other.

This gave an easy way in to identifying a set of folk who had expressed an interest in a link by virtue of sharing it, this set then acting as the starting point for a community analysis.

Another approach I started to explore (but never blogged?!) was looking at networks of folk who had shared one of the links recently shortened by a particular bit.ly user. So for example, this graph (captured some time ago) used the BackType API to find who had tweeted one of more of 15 or so links that @charlesarthur had shortened using bit.ly, and then plotted friend connections between them:

Unfortunately, now that the BackType API has gone (when I try to call it I get a “Limit exceeded” error message), the key ingredient from those two original recipes is no longer available…:-(

Twitter may lay claim to millions of users, but we intend to only inhabit a small part of it… I Follow 500 or so people, and am followed by 3000 or so “curated” followers (I block maybe 20-50 a week, and not all of them obvious spam accounts, in part because I see the list of folk who follow me as a network that defines several particular spheres of interest, and I don’t want to drown out signal by noise.)

So here’s a map of how the connected component of the graph of how my Twitter followers follow each other; it excludes people who aren’t followed by anyone in the graph (which may include folk who do follow me but who have private accounts).

The layout is done in Gephi using the Force Atlas 2 layout. It’s quite by chance that the layout resembles the Isle of Wight…or a heart? Yes, maybe it’s a great be heart:-)

By running the HITS statistic over the graph, we can get a feel for who the influential folk are; sizing and labeling nodes by Authority, we get this (click through to see a bigger version):

Here’s an annotated version, as I see it (click through to see a bigger version):

If you’d like me to spend up to 20 mins making a map for you, I’ll pick up on an idea from Martin Hawksey and maybe do two or three maps for a donation to charity (in particular, to Ovacome). In fact, if you feel as if you’ve ever benefited from anything posted to this blog, why not give them a donation anyway…? Donate to ovacome.

Given absolutely no-one picked up on List Intelligence – Finding Reliable, Trustworthy and Comprehensive Topic/Sector Based Twitter Lists, here’s a example of what the technique might be good for…

Seeing the tag #edusum11 in my feed today, and not being minded to follow it it I used the list intelligence hack to see:

- which lists might be related to the topic area covered by the tag, based on looking at which Twitter lists folk recently using the tag appear on;
- which folk on twitter might be influential in the area, based on their presence on lists identified as maybe relevant to the topic associated with the tag…

Here’s what I found…

Some lists that maybe relate to the topic area (username/list, number of folk who used the hashtag appearing on the list, number of list subscribers), sorted by number of people using the tag present on the list:

/joedale/ukedtech 6 6
/TWMarkChambers/edict 6 32
/stevebob79/education-and-ict 5 28
/mhawksey/purposed 5 38
/fosteronomo/chalkstars-combined 5 12
/kamyousaf/uk-ict-education 5 77
/ssat_lia/lia 5 5
/tlists/edtech-995 4 42
/ICTDani/teched 4 33
/NickSpeller/buzzingeducators 4 2
/SchoolDuggery/uk-ed-admin-consultancy 4 65
/briankotts/educatorsuk 4 38
/JordanSkole/jutechtlets 4 10
/nyzzi_ann/teacher-type-people 4 9
/Alexandragibson/education 4 3
/danielrolo/teachers 4 20
/cstatucki/educators 4 13
/helenwhd/e-learning 4 29
/TechSmithEDU/courosalets 4 2
/JordanSkole/chalkstars-14 4 25
/deerwood/edtech 4 144

Some lists that maybe relate to the topic area (username/list, number of folk who used the hashtag appearing on the list, number of list subscribers), sorted by number of people subscribing to the list (a possible ranking factor for the list):
/deerwood/edtech 4 144
/kamyousaf/uk-ict-education 5 77
/SchoolDuggery/uk-ed-admin-consultancy 4 65
/tlists/edtech-995 4 42
/mhawksey/purposed 5 38
/briankotts/educatorsuk 4 38
/ICTDani/teched 4 33
/TWMarkChambers/edict 6 32
/helenwhd/e-learning 4 29
/stevebob79/education-and-ict 5 28
/JordanSkole/chalkstars-14 4 25
/danielrolo/teachers 4 20
/cstatucki/educators 4 13
/fosteronomo/chalkstars-combined 5 12
/JordanSkole/jutechtlets 4 10
/nyzzi_ann/teacher-type-people 4 9
/joedale/ukedtech 6 6
/ssat_lia/lia 5 5
/Alexandragibson/education 4 3
/NickSpeller/buzzingeducators 4 2
/TechSmithEDU/courosalets 4 2

Other ranking factors might include the follower count, or factors from some sort of social network analysis, of the list maintainer.

Having got a set of lists, we can then look for people who appear on lots of those lists to see who might be influential in the area. Here’s the top 10 (user, number of lists they appear on, friend count, follower count, number of tweets, time of arrival on twitter):

['terryfreedman', 9, 4570, 4831, 6946, datetime.datetime(2007, 6, 21, 16, 41, 17)]
['theokk', 9, 1564, 1693, 12029, datetime.datetime(2007, 3, 16, 14, 36, 2)]
['dawnhallybone', 8, 1482, 1807, 18997, datetime.datetime(2008, 5, 19, 14, 40, 50)]
['josiefraser', 8, 1111, 7624, 17971, datetime.datetime(2007, 2, 2, 8, 58, 46)]
['tonyparkin', 8, 509, 1715, 13274, datetime.datetime(2007, 7, 18, 16, 22, 53)]
['dughall', 8, 2022, 2794, 16961, datetime.datetime(2009, 1, 7, 9, 5, 50)]
['jamesclay', 8, 453, 2552, 22243, datetime.datetime(2007, 3, 26, 8, 20)]
['timbuckteeth', 8, 1125, 7198, 26150, datetime.datetime(2007, 12, 22, 17, 17, 35)]
['tombarrett', 8, 10949, 13665, 19135, datetime.datetime(2007, 11, 3, 11, 45, 50)]
['daibarnes', 8, 1592, 2592, 7673, datetime.datetime(2008, 3, 13, 23, 20, 1)]

The algorithms I’m using have a handful of tuneable parameters, which means there’s all sorts of scope for running with this idea in a “research” context…

One possible issue that occurred to me was that identified lists might actually cover different topic areas – this is something I need to ponder…

Have you ever found yourself in the situation where you want to send someone a Twitter message but you can’t remember their Twitter username although you do know their real name? Or where you can remember their twitter username or their real name, but you do remember who they work for, or some other biographical fact about them that might appear in their Twitter biography? If that sounds familiar, here’s a trick that may help…

… a searchable Twitter friends and followers contact list in Google Spreadsheets.

It’s based on Martin Hawksey’s rather wonderful Export Twitter Followers and Friends using a Google Spreadsheet (I have to admit – Martin has left me way behind now when it comes to tinkering with Google Apps Script…!) To get started, you’ll need a Google docs account, and then have to indulge in a quick secret handshake between Google docs and Twitter, but Martin’s instruction sheet is a joy to follow:-) Follow the *** Google Spreadsheet to Export Twitter Friends and Followers *** link on Martin’s page, then come back here once you’ve archived your Twitter friends and/or followers…

..done that? Here’s how to make the contact list searchable… I thought it should have been trivial, but it turned out to be quite involved!

The first thing I did was create a drop down list to let the user select Friends or Followers as the target of the search. (Martin’s application loads friends and followers into different sheets.)

The next step was to generate a query. To search for a particular term on a specified sheet we can use a QUERY formula that takes the following form:

=query(Friends!B:E,”select B,C,D,E where D contains ‘JISC’”)

Friends! specifies the sheet we want to search over; B:E says we want to pull columns B, C, D and E from the Friends sheet into the current sheet; the select statement will display results over four columns (B, C, D and E) from Friends for rows where the entry in column D contains the search term JISC.

To pull in the search term from cell D1 we can use a query of the form:

=query(Friends!B:E,concatenate(“select B,C,D,E where D contains ‘”,D1,”‘”))

The =concatenate formula constructs the search query. Make sure you use the right sort of quotes when constructing the string – Google Spreadsheets seems to prefer the use of double quotes wherever possible!

To search over two columns, (for example, the real name and the description columns of the twitter friends/follower data) we can use a query of the form:

=query(Followers!B:E,concatenate(“select B,C,D,E where C contains ‘”,D1,”‘ or D contains ‘”,D1,”‘”)

Again – watch out for the quotes – the result we want from the concatenation is something like:

=query(Followers!B:E,concatenate(“select B,C,D,E where C contains ‘Jisc’ or D contains ‘Jisc’)

so we have to explicitly code in the single quote in the concatenation formula.

Unfortunately, the query formula is case sensitive, which can cause the search to fail because we haven’t taken (mis)use of case into account in our search term. This means we need to go defensive in the query formulation – in the following example, I force everything to upper case – search corpus as well as search terms:

=query(Followers!B:E,concatenate(“select B,C,D,E where upper(C) contains upper(‘”,D1,”‘) or upper(D) contains upper(‘”,D1,”‘)”)

The final step is to define the sheet we want to search – Friends! or Followers! – depending on the setting of cell B1 in our search sheet. I had idly though I could use a concatenate formula to create this, but concatenate returns a string and we need to define a range. In the end, the workaround I adopted was an if statement, that chooses a query with an appropriate range set explicitly/hardwired within the formula depending on whether we are are searching Friends or Followers. Here’s the complete formula, which i put into cell E1.

=if(B1=”Friends”,query(Friends!B:E,concatenate(“select B,C,D,E where upper(C) contains upper(‘”,D1,”‘) or upper(D) contains upper(‘”,D1,”‘)”)),query(Followers!B:E,concatenate(“select B,C,D,E where upper(C) contains upper(‘”,D1,”‘) or upper(D) contains upper(‘”,D1,”‘)”)))

I now have a query sheet defined that allows me to search over my friends or followers, as required, according to their real name or a search term that appears in their biography description.

I’ve been having a play with Twitter again, looking at how we can do the linked thing without RDF, both within a Twitter context and also (heuristically) outside it.

First up, hashtag discovery from Twitter lists. Twitter lists can be used to collect together folk who have a particular interest, or be generated from lists of people who have used a particular hashtag (as Martin H does with his recipe for Populating a Twitter List via Google Spreadsheet … Automatically! [Hashtag Communities]).

The thinking is simple: grab the most recent n tweets from the list, extract the hashtags, and count them, displaying them in descending order. This gives us a quick view of the most popular hashtags recently tweeted by folk on the list: Popular recent tags from folk a twitter list

This is only a start, of course: it might be that a single person has been heavily tweeting the same hashtag, so a sensible next step would be to also take into account the number of people using each hashtag in ranking the tags. It might also be useful to display the names of folk on the list who have used the hashtag?

I also updated a previous toy app that makes recommendations of who to follow on twitter based on a mismatch between the people you follow (and who follow you) and the people following and followed by another person – follower recommender (of a sort!):

The second doodle was inspired by discussions at Dev8D relating to a possible “UK HE Developers’ network”, and relies on an assumption – that the usernames people use Twitter might be used by the same person on Github. Again, the idea is simple: can we grab a list of Twitter usernames for people that have used the dev8d hashtag (that much is easy) and then lookup those names on Github, pulling down the followers and following lists from Github for any IDs that are recognised in order to identify a possible community of developers on Github from the seed list of dev8d hashtagging Twitter names. (It also occurs to me that we can pull down projects Git-folk are associated with (in order to identify projects Dev8D folk are committing to) and the developers who also commit or subscribe to those projects.)

As the above network shows, it looks like we get some matches on usernames…:-)

Reading @briankelly’s post on Institutional Use of Twitter by Russell Group Universities just now, where he refers to an old list of Twitter accounts compiled (in a blog post) by Liz Azyan, I wondered how I’d go about finding, or compiling, a comprehensive list of official Twitter accounts for UK HE institutions.

My first thought was: Google “uk universities twitter list” to find lists folk are currently curating themselves. This search (for me) turns up lists such as @thirdyearabroad/uk-universities and @bellerbys/uk-universities/, as well as Liz’s post.

Peeking at the two lists mentioned above, I notice they follow different numbers of Twitter users, which suggests to me I need to build a couple of scripts:

1) a script that will take a list of N Twitter lists and generate a union list
2) a script that will take a list of N Twitter lists and generate the set of users who:
2a) appear on each list;
2b) appear on at least M of N lists;
2c) appear on only one list.

The second idea that came to mind was in response to the question: is there a more effective way of finding lists of UK HE Twitter accounts? And what came to mind was this script I now need to hack together:

3) pick a handful of (official UK university) Twitter account IDs and pull down the lists that they have been added to; find the intersection set of lists that follow all the users in the test set, under the assumption that these may be lists that follow some common characteristic of those users, e.g. the fact that they are official UK university accounts.

This second approach uses a set of user IDs you would expect to be on a particular list as bait for finding lists that do include all those IDs.

(A corollary of this might be to look at a set of people you might expect to follow a particular sort of list, pull down the lists they follow/subscribe to, and then look for the intersection set of lists that all the sample users follow…?)

The third idea was to see whether I could find Twitter accounts linked to from university pages, searching Google for things like site:ac.uk link:twitter.com, but that wasn’t very satisfactory. Searching for university site:twitter.com improves matters, and inspection of the result suggests intitle:university site:twitter.com may be even more effective…

Searching for intitle:university site:twitter.com -inurl:status excludes results from tweets, but we still get results relating to lists as well as accounts. I can’t find an obvious way of only searching for UK universities…

So the exercise is this: how would you generate a comprehensive list of official UK university Twitter accounts?

PS FWIW, I think coming up with search/discovery strategies such as the above is the sort of information skill we might consider or indeed, expect to be a graduate level information skill. Discuss.

PPS I would of course appreciate ideas for alternative, and indeed, more effective ways of completing the “how to find lists of official Twitter accounts for UK universities” in the comments:-)

Yesterday morning, @ambrouk tweeted: “Is there a tool where you can quickly check 2 twitter accs and find out who follows both? I.e who can see the exchanges between A and B?”

(Explanatory note: if you start a tweet with @name, only your twitter followers who also follow @name will see the tweet in their stream.)

There probably is, but I thought it’d be an interesting exercise to see if I could put a script together to do this in a web page without requiring authenticated access to the Twitter API using the Google Social Graph API.

If you go to the Social Graph API Parameter Playground, you can use the tool provided to construct API calling URLs that return the people you follow on Twitter, or who follow you, as well as various other bits of social data…

The data is returned as JSON, so it’s easy enough to pull into a web page. So here’s a view of my ‘common twitter friends’ single page web app:

And here’s the script that does it, pulling back the common friends or common followers of two folks on Twitter:

<html><head><title></title>

<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js"></script> 
 
<script type="text/javascript">

function compareUsers(typ){
  if (typ=='followers'){
    gtyp='edi'
  } else {
    gtyp='edo'
  }
  url='http://socialgraph.apis.google.com/lookup?q=http://twitter.com/'+$('#user1').val()+',http://twitter.com/'+$('#user2').val()+'&'+gtyp+'=1&callback=?';

  var content = "";
  $.getJSON(url,
   function(json){
     if(json) {
       if (typ=='followers')
         content = commonFollowers(json);
       else 
         content = commonFriends(json);
     } else {
       content = "The request did not return results.";
     }
     var list=''
     for (i in content)
       if (content[i].indexOf('http://twitter.com/account/redirect_by_id')==-1)
     	list+=" "+content[i].replace('http://twitter.com/','')
     $("#output").html(list);
     $("#u1p").html('of which '+content.length+' in common')
     $("#u2p").html('of which '+content.length+' in common')
   }
  );
}

function commonFriends(json){
  return commonXs(json,'friends')
}

function commonFollowers(json){
  return commonXs(json,'followers')
}

function commonXs(json,typ){
  var name=new Array()
  if (typ=='followers')
    nref='nodes_referenced_by'
  else
    nref='nodes_referenced'
  Xers=new Array();
  coXs=new Array();

  for (u in json['nodes']) {
    name[name.length]=u.replace('http://twitter.com/','')
	lXers=new Array()
	for (i in json['nodes'][u][nref])
		lXers[lXers.length]=i	
	Xers[Xers.length]=lXers
  }
  $("#u1n").html(name[0]+': ')
  $("#u1nn").html(' with '+name[0])
  $("#u1c").html(Xers[0].length+' '+typ)
  $("#u2n").html(name[1]+': ')
  $("#u2nn").html(' with '+name[1])
  $("#u2c").html(Xers[1].length+' '+typ)
  for (i in Xers[0]){
    if (Xers[1].indexOf(Xers[0][i])>-1){
	  coXs[coXs.length]=Xers[0][i]
	}
  }
  return coXs
}
</script>

</head>

<body>
<form>
User 1: @<input type='text' id='user1' value="ambrouk" />
User 2: @<input type='text' id='user2' value='psychemedia' />
<input type='button' value='Find Common Followers' onclick='compareUsers("followers")' /> <input type='button' value='Find Common Friends' onclick='compareUsers("friends")' />
</form>
<div><span id='u1n'></span> <span id='u1c'></span> <span id='u1p'></span> <span id='u2nn'></div>
<div><span id='u2n'></span> <span id='u2c'></span> <span id='u2p'></span> <span id='u1nn'></div>
<hr/>
<div id="output"></div>

</body></html>

With a bit of tweaking, it should be possible to generalise the code to find the common followers of N friends (please post a link to the script in a comment if you do this:-)

Rather than me hosting the script, you’ll have to run it yourself. Save the raw code from here as an .html file on your desktop, then drag it and drop it into a browser window (or more laboriously, open the file you saved from the browser file menu).

[UPDATE: try it out here - Common Twitter friends]

PS I’m not sure that the limits are on the Google Social Graph API, whether in terms of the number of accounts that can be included in a single query, the size of the returned payload from a single query, or the number of queries per minute/hour/day, so I’m not sure if this represents a way round the Twitter API limit if all you’re doing is calling on friends and followers lists?

I’m now following about 500 or so people on Twitter, but to what extent are they following each other? Are there any noticeable subgroups in the folk I follow, by virtue of them being highly linked to each other in the friends and following stakes?

Each of the nodes represents one of my Twitter friends (that is, each node represents a separate person I follow on Twitter).

Node size is proportional to the number of my friends who are following other of my friends.

Node colour is proportional the the number of my friends that person is following (blue is cold – low number; red is hot – high number).

The graph is an indication of the extent to which the people I follow (that is, my friends…) is an echo chamber…

Running the Gephi “connected cpmponents” statistic, it seems that the group is pretty tightly connected… There is one noticeable separate component that contains more than a singleton, from a few accounts I followed last year…:

If I look at the labels for the other separate components (not shown), they mainly correspond to people with private accounts, although there are a couple of people who are completely independent of the rest of my Twitter social circle.

The Gephi modularity class statistic, however, suggests there is a little more structure hiding in there…

(This is a random algorithm, so it may give slightly different answers each time it is run…)

Let’s peek inside them…

Looks a bit educationalist to me…;-)

How about this one:

Hmm. Government and open data, maybe? What next…?

BBC and journ hack types, with a bit of datajourn thrown in maybe?

Hmmm – the next one looks like an OU cluster:

And that leaves….

JISC, museums and libraries…

Seems about right to me:-)

PS Images produced using Gephi… Note to self: start spending a ittle more time about tidying up the presentation of some of these images…;-)

PPS for a similar exercise applied to my Facebook friends, see Getting Started With The Gephi Network Visualisation App – My Facebook Network, Part IV

A couple of weeks ago, I put started cobbling together some clunky scripts to collate network data files from lists of people twittering with a particular hashtag (First Glimpses of the OUConf10 Hashtag Community). I’ve got a Twapperkeeper key now, so the next step is to pull archived hashtagged tweets from there to generate my hashtaggers list, and then use that data as the basis for pulling in friends and followers links for particular individuals from the Twitter API.

One thing I’d like to start pulling together is a set of tools for providing network and backchannel analysis around hashtag communities. Andy Powell has already published a site that summarises hashtag activity in the form of Summarizr using a Twapperkeeper archive:

So what else might we look for?

Mulling over my own Personal Twitter Networks in Hashtag Communities, the metrics I report include:

- Number of hashtaggers [Ngalaxy]
- Hashtaggers as followers (‘hashtag followers’) [Gfollowers]
- Hashtaggers as friends (‘hashtag friends’) [Gfriends]
- Hashtagger followers not friended (‘serfs’) [Gserfs]
- Hashtagger friends not following (‘slebs’) [Gslebs]
- Hashtaggers not friends or followers (‘the hashtag void’) [Gvoid]
- Reach into hashtag community [Greach=Gfollowers/Ngalaxy]
- Reception of hashtag community the proportion of the the hashtag community that are followed by (i.e. are friends of) the named individual; [Greception=Gfriends/Ngalaxy]
- Hashtag void (normalised) [Normvoid=Gvoid/Ngalaxy]
- Total personal followers the total number of followers of the named individual [Nfollowers]
- Total personal friends: the total number of friends of the named individual [Nfriends]
- Hashtag community dominance of personal reach: the extent to which the hashtag community dominates the set of people who follow the named individual, [Domreach=Gfollowers/Nfollowers]
- Hashtag community dominance of personal reception: the extent to which the set of the named individual’s friends is dominated by members of the hashtag community, [Domreception=Gfriends/Nfriends]

Anyway, it strikes me that calculating those measures as means (and standard deviations) across all the members of the network, along with more traditional social network analysis network centrality or clustering measures, might help identify different signatures relating to the maturity of different hashtag communities (for example, the extent to which they are just forming, or the extent to which they have largely saturated in terms of members knowing each other).

These metrics might also change over the course of an event being discussed via a particular hashtag.

Last week, I got an email from Stuart with the following query:

I’m trying to find a way to enable people to post to the OU twitter account from their personal account by using a predefined hashtag. …
We agreed a hashtag #***** which kmi researchers are using from their account if they want to share information to the main OU account.  I pull an RSS feed of this into the OU account and retweet it.  I’m sure you can see the obvious loop that occurs!

… are you aware of anything that will let me retweet a hashtag and strip off that hashtag to avoid the loop?  It would be great to be able to add new hashtags in the future so it could be rolled out to other faculties who might wish to share their news via the OU account just by tweeting from individual faculty members’ accounts.

Here’s what I came up with…

The first part of the pipe takes the user defined hashtag and creates the URL that will run a search for that hashtag on twitter and the second part of the pipe fetches the feed. The Filter block will only pass through tweets that come from specified twitter users (actually, that isn’t quite true… this pipe is gameable/spammable becuase of the way I use “contains” in the whitelist filter block… Can you see how?!;-) The regular expression block strips the hashtag out of the retweeted tweets. (For the pipe to work and not get into an infinite loop, this isn’t actually necessary if we’re using the whitelist, because retweeters that make use of the pipe feed should not have their username in the whitelist… That is, if you’re running the whitelist, you can remove the regualr expression block and leave the hashtag in the retweet feed. Conversely, if you don’t want to run the whitelist, you can just remove the filter block, although in this case you will need the hashtag stripping regular expression block to prevent infinite retweets… Got that?!;-)

You can find the pipe here: Hashtag retweeter pipe

If you want a more “secure” version, i.e. one that does not reveal the identities of people in the whitelist, or the hashtag, use private string blocks (example pipe:

If you want to create your own hashtag retweeter pipe without having to clone and customise your own pipe, use this approach:

(NB if you leave either of the username slots blank, then tweets sent by anyone using the hastag will be passed through the pipe and made available for retweeting.)

Sigh… another claim… 2ZXZGU4TDXK2