Tag: hashtag

Who’s Tweeting Our Hashtag?

Last night, I put togther a quick video showing how to make a Yahoo Pipe that will find who’s been tweeting with a particular hashtag, in particular using the opened09 hashtag:

Here’s the pipe, deconstructed:

The idea behind this pipe in part came from a Dave Winer utility that will create an OPML feed linking to the Twitter feeds from the people that a named Twitter user is following (linked to from rssCloud news); my immediate reaction to that was: would an OPML feed linking to the feeds of everyone who had used a particular hashtag be useful? and this pipe is the result.

(Note to self: I really, really, really need to put a Pipes2OPML script together…)

Anyway, I thought it might make sense to generalise this pipe, and hopefully make it a little more useful, as well as showing a Pipes trick or two in some sort of appropriate context. So here are some problems with the above pipe that I’ll then show you how to fix…

– the pipe is hardwired to search for tweets containing #opened09; how can w generalise it to work with other hashtags/search strings?
– limited number of search results: the pipe only has access to the one hundred most recent opened09 hashtagged tweets; how can we get access to more?
– for a pipe that shows who has been using a particular hashtag, it might also be useful to see how many times they have used that hashtag?
– the pipe displays the hashtag twitterers in the order in which they first tweeted (a history based ordering); which is useful in one sense, but not in another. How else might we order the tweets? Recency of posting? Most active user of the hashtag?

So, first up, how do we generalise the pipe to work with other search terms? Simple, just add a user input for the search term and construct a URI using that term; then wire the URI in to the Fetch Feed block:

A form element appears on the pipes front page, and the search term is also passed via URIs that call the pipe:

Secondly, how do we increase the pool of hashtagged tweets? Simple: Twitter uses paged search results so we can pull in results from those other pages too…

So we can construct URIs for the second, third, fourth etc pages of search results too, and pull feeds in from all those pages:

The third item on our list was displaying how many times each user in the list had used the hashtag (based on the number of times they appeared in the search results sample, of course).

As @ostephens originally pointing out to me (at MashLib09), the Unique block actually counts the number of occurrences of the unique filter term:

We can display this in the title element using a very powerful construction… In the regular expression, rewrite the title string containing the value of the repeatcount variable. How? Like this – access the variable using the construction: ${VARIABLE_NAME}

So in the current example, we can get the value of the count using ${y:repeatcount}:

The final item on our improvements list was to try out some other orderings of the displayed tweets. At the moment, they are ordered according to the date on which each user first used the hashtag term (that is, within the sample of tweets we have pulled back from the Twitter search).

To change the order of names that are output from the pipe, we can use the Sort block. So for example, sort the users in terms of activity:

To sort the users in reverse chronological order (that is, according to who used the hashtag most recently), sort on the published attribute rather than on y:repeatcount.

You can now grab the RSS feed from the pipe and consume it in anything that accepts RSS, pull the JSON feed into your own web page if you’re a little more adventurous, or grab a Google homepage widget from the pipe’s homepage: Hashtag Twitterers Pipe

To tune the pipe to your own needs, you can also clone it from there and then modify your own version of it to your heart’s content:-)

Searching for Twitter Hashtags and Finding Hashtag Communities

Over the last few weeks I’ve been messing around more than I should with Twitter, and in particular trying to get a feel for how we might use hashtag communities as a well of identifying and growing community structures in a particular topic area (see posts all over OUseful.info for more details).

A couple of days ago, @clarileia raised the question of how you find new hashtags, so I had a little tinker today putting together a couple of hacks (Twitter hashtag search pipe and Twitter my network hashtags) that let you identify recently used Twitter hashtags associated with a particular search term, or with a specified user’s recent friends or followers.

Twiitter hashtag search http://pipes.yahoo.com/ouseful/twitterhashtagsearch

[Note: at the time of writing, Pipes appears to be running a little slow… if the Pipe appears to stall, it does work, honest… try it again later ;-)]

At the core of all the hacks is a clunky hashtag tokeniser pipe that takes a Twitter status update and pulls out the hashtags:

This utility pipe works by taking the status update, extracting the hashtags using a hacked together regular expression, splits the separate hashtags into separate feed items, and then filters them to emit only legitimate hashtags.

The utility pipe is then used in a search powered pipe, which searches twitter for the 100 most recent tweets containing the search terms and then scans those for hashtags; and a ‘personal network hashtags’ pipe that takes a Twitter username, pulls back the tweets from their one hundred most recent friends, and their one hundred most recently followers, and then scans those tweets for hashtags.

For example, here’s the search pipe:

Both pipes have a common output routine – the list of hashtags is filtered through the Unique block, which also returns a count of how many times each hashtag has appeared. The hashtags are then ordered and filtered according to the minimum number of required occurrences in the sample. A regular expression adds the number of occurrences of each hashtag.

The pipes could be extended to pull in more search results, or more followers/friends (maybe the first hundred friends/followers as well as the most recent hundred?) but that’s left as an exercise for the reader. As for the use case – I dunno? Maybe integration with the OUseful TwitterMyHashtag apps? Or perhaps @clarileia had a use case in mind?!;-)

PS thanks to PJ on the Yahoo Pipes team for getting back to me earlier today when I was struggling with a slow running pipes editor… I’m now totally reliant on Pipes for many apps, and especially for rapid the majority of my prototyping, so when Pipes is slow, I feel as happy as if I’ve lost an unbacked up server… Brian Kelly would probably tell me I need to do a risk assessment… I’ve already done one: What Happens If Yahoo! Pipes Dies? – but I haven’t made a start on the contingency stuff that was considered there…