Yahoo is laying off again, so just in case, if you’re a user of Yahoo Pipes, it may be worth exporting the “source code” of your pipes or any pipes that you make frequent use of in case the Yahoo Pipes service gets cut.
Why? Well, a little known fact about Yahoo pipes is that you can get hold of a JSON representation from a pipe that describes how the pipe is constructed…
…and some time ago, Greg Gaughan started working on a script that allows you to “compile” these descriptions of your Yahoo Pipes into Python programming code that can be run as a standalone programme on your own server: Pipe2Py. (Greg also did a demo that allowed Pipes to be “migrated” to a version of Pipe2Py running on Google App Engine.)
From a quick skim over the Pipes service, it seems you can get hold of a list of published pipes for a user easily enough, which means we can get a quick dump of the “source code” of all the published pipes for a given user (and then maybe compile them to Python using pipe2py so we can essentially keep the functionality running…). Here’s a first pass at a bulk exporter: Yahoo Pipes Exporter (published pipes by user).
To get a full list of pipes by user, I think you need to be logged in as that user?
See also: Yahoo Pipes Code Generator (Python): Pipe2Py
PS do I need to start worrying about flickr too?!
There was a recent discussion on the Google Refine group about exporting Refine GREL operations which also are stored in a JSON representation to R Studio (the idea is you design your data manipulation in Refine to use in R) http://groups.google.com/group/google-refine/browse_thread/thread/e5502cba4ade1075.
Your post makes me wonder if a Yahoo Pipe JSON to Google Refine JSON or Pipe to R description would be useful (with the later cloudstat.org into play) … hmm now I’m thinking if Pipe2Py is making a python representation of a Pipe then maybe it could be a great addition to scraperwiki.com … maybe?
Martin
@martin the Pipe2Py library is available on Scraperwiki, but I haven’t yet tried get it working… I’m not sure if pipe2py wrote stuff to and read stuff from saved files? If it does, then a tweak or two may be required to get it to work instead with the db? I’ll try to have a play this week – unless you beat me to it, of course…;-)
UPDATE: here’s a first attempt at getting pipe2py running in scraperwiki, along with a couple thoughts in the src about what the next step(s) might be… https://scraperwiki.com/scrapers/pipe2py_test/
the list for me doing something in scraperwiki is now over an arm long. I’ve also hopefully got my evenings back so I might even get a chance to play.