OUseful.Info, the blog…

Trying to find useful things to do with emerging technologies in open education

Just in Case – Saving Your Yahoo Pipes…

Yahoo is laying off again, so just in case, if you’re a user of Yahoo Pipes, it may be worth exporting the “source code” of your pipes or any pipes that you make frequent use of in case the Yahoo Pipes service gets cut.

Why? Well, a little known fact about Yahoo pipes is that you can get hold of a JSON representation from a pipe that describes how the pipe is constructed…

…and some time ago, Greg Gaughan started working on a script that allows you to “compile” these descriptions of your Yahoo Pipes into Python programming code that can be run as a standalone programme on your own server: Pipe2Py. (Greg also did a demo that allowed Pipes to be “migrated” to a version of Pipe2Py running on Google App Engine.)

From a quick skim over the Pipes service, it seems you can get hold of a list of published pipes for a user easily enough, which means we can get a quick dump of the “source code” of all the published pipes for a given user (and then maybe compile them to Python using pipe2py so we can essentially keep the functionality running…). Here’s a first pass at a bulk exporter: Yahoo Pipes Exporter (published pipes by user).

To get a full list of pipes by user, I think you need to be logged in as that user?

See also: Yahoo Pipes Code Generator (Python): Pipe2Py

PS do I need to start worrying about flickr too?!

Written by Tony Hirst

April 10, 2012 at 2:24 pm

Posted in Anything you want

Tagged with

5 Responses

Subscribe to comments with RSS.

  1. There was a recent discussion on the Google Refine group about exporting Refine GREL operations which also are stored in a JSON representation to R Studio (the idea is you design your data manipulation in Refine to use in R) http://groups.google.com/group/google-refine/browse_thread/thread/e5502cba4ade1075.

    Your post makes me wonder if a Yahoo Pipe JSON to Google Refine JSON or Pipe to R description would be useful (with the later cloudstat.org into play) … hmm now I’m thinking if Pipe2Py is making a python representation of a Pipe then maybe it could be a great addition to scraperwiki.com … maybe?

    Martin

    mhawksey

    April 10, 2012 at 2:47 pm

    • @martin the Pipe2Py library is available on Scraperwiki, but I haven’t yet tried get it working… I’m not sure if pipe2py wrote stuff to and read stuff from saved files? If it does, then a tweak or two may be required to get it to work instead with the db? I’ll try to have a play this week – unless you beat me to it, of course…;-)

      Tony Hirst

      April 10, 2012 at 2:58 pm

    • the list for me doing something in scraperwiki is now over an arm long. I’ve also hopefully got my evenings back so I might even get a chance to play.

      mhawksey

      April 10, 2012 at 3:13 pm

  2. [...] with it. So today, I thought I’d better address that. Building on the script I linked to from Just in Case – Saving Your Yahoo Pipes…, I put together a simple Scraperwiki script that grabs the JSON descriptions of my public/published [...]


Comments are closed.

Follow

Get every new post delivered to your Inbox.

Join 787 other followers

%d bloggers like this: