Yahoo Pipes Code Generator (Python): Pipe2Py

Wouldn’t it be nice if you coud use Yahoo Pipes as a visual editor for generating your own feed powered applications running on your own server? Now you can…

One of the concerns occasionally raised around Yahoo Pipes (other than the stability and responsiveness issues) relates to the dependence that results on the Yahoo pipes platform from creating a pipe. Where a pipe is used to construct an information feed that may get published on an “official” web page, users need to feel that content will always be being fed through the pipe, not just when when Pipes feels like it. (Actually, I think the Pipes backend is reasonably stable, it’s just the front end editor/GUI that has its moments…)

Earlier this year, I started to have a ponder around the idea of a Yahoo Pipes Documentation Project (the code appears to have rotted unfortunately; I think I need to put a proper JSON parser in place:-(, which would at least display a textual description of a pipe based on the JSON representation of it that you can access via the Pipes environment. Around the same time, I floated an idea for a code generator, that would take the JSON description of a pipe and generate Python or PHP code capable of achieving a similar function to the Pipe from the JSON description of it.

Greg Gaughan picked up the challenge and came up with a Python code generator for doing just that, written in Python. (I didn’t blog it at the time because I wanted to help Greg extend the code to cover more modules, but I never delivered on my part of the bargain:-(

Anyway – the code is at http://github.com/ggaughan/pipe2py and it works as follows. Install the universal feed parser (sudo easy_install feedparser) and simplejson (sudo easy_install simplejson), then download Greg’s code and declare the path to it, maybe something like:
export PYTHONPATH=$PYTHONPATH:/path/to/pipe2py.

Given the ID for a pipe on Yahoo pipes, generate a Python compiled version of it:
python compile.py -p PIPEID

This generates a file pipe_PIPEID.py containing a function pipe_PIPEID() which returns a JSON object equivalent of the output of the corresponding Yahoo pipe, the major difference being that it’s the locally compiled pipe code that’s running, not the Yahoo pipe…

So for example, for the following simple pipe, which just grabs the OUseful.info blog feed and passes it straight through:

SImple pipe for compilation

we generate a Python version of the pipe as follows:
python compile.py -p 404411a8d22104920f3fc1f428f33642

This generates the following code:

from pipe2py import Context
from pipe2py.modules import *

def pipe_404411a8d22104920f3fc1f428f33642(context, _INPUT, conf=None, **kwargs):
    "Pipeline"
    if conf is None:
        conf = {}

    forever = pipeforever.pipe_forever(context, None, conf=None)

    sw_502 = pipefetch.pipe_fetch(context, forever, conf={u'URL': {u'type': u'url', u'value': u'https://blog.ouseful.info/feed'}})
    _OUTPUT = pipeoutput.pipe_output(context, sw_502, conf={})
    return _OUTPUT

We can then run this code as part of our own program. For example, grab the feed items and print out the feed titles:

context = Context()
p = pipe_404411a8d22104920f3fc1f428f33642(context, None)
for i in p:
  print i['title']

running a compiled pipe on the desktop

Not all the Yahoo Pipes blocks are implemented (if you want to volunteer code, I’m sure Greg would be happy to accept it!;-), but for simple pipes, it works a dream…

So for example, here’s a couple of feed mergers and then a sort on the title…

ANother pipe compilation demo

And a corresponding compilation, along with a small amount of code to display the titles of each post, and the author:

from pipe2py import Context
from pipe2py.modules import *

def pipe_2e4ef263902607f3eec61ed440002a3f(context, _INPUT, conf=None, **kwargs):
    "Pipeline"
    if conf is None:
        conf = {}

    forever = pipeforever.pipe_forever(context, None, conf=None)

    sw_550 = pipefetch.pipe_fetch(context, forever, conf={u'URL': [{u'type': u'url', u'value': u'https://blog.ouseful.info/feed'}, {u'type': u'url', u'value': u'http://feeds.feedburner.com/TheEdTechie'}]})
    sw_572 = pipefetch.pipe_fetch(context, forever, conf={u'URL': {u'type': u'url', u'value': u'http://www.greenhughes.com/rssfeed'}})
    sw_580 = pipeunion.pipe_union(context, sw_550, conf={}, _OTHER = sw_572)
    sw_565 = pipesort.pipe_sort(context, sw_580, conf={u'KEY': [{u'field': {u'type': u'text', u'value': u'title'}, u'dir': {u'type': u'text', u'value': u'ASC'}}]})
    _OUTPUT = pipeoutput.pipe_output(context, sw_565, conf={})
    return _OUTPUT


context = Context()
      
p = pipe_2e4ef263902607f3eec61ed440002a3f(context, None)
for i in p:
        print i['title'], ' by ', i['author']

And the result?
MCMT013:pipes ajh59$ python basicTest.py
Build an app to search Delicious using your voice with the Android App Inventor by Liam Green-Hughes
Digging Deeper into the Structure of My Twitter Friends Network: Librarian Spotting by Tony Hirst
Everyday I write the book by mweller
...

So there we have it.. Thanks to Greg, the first pass at a Yahoo Pipes to Python compiler…

PS Note to self… I noticed that the ‘truncate’ module isn’t supported, so as it’s a relatively trivial function, maybe I should see if I can write a compiler block to implement it…

PPS Greg has also started exploring how to export a pipe so that it can be run on Google App Engine: Running Yahoo! Pipes on Google App Engine

17 comments

  1. Pingback: newid Yahoo Pipes i Python | Hacio'r Iaith
  2. Pingback: I’ve been reading… | andydickinson.net
  3. Michael Kowalchik

    Tony, very cool, I like it. :) I especially like the idea of giving people an “upgrade path” when pipes can’t cut it anymore. That was one of the biggest limitations I saw with pipes from the beginning, makes easy stuff easy, hard stuff impossible. This gives people a way to get to the hard stuff.

  4. Tony Hirst

    Some random thoughts I just emailed to Greg, who built pipe2py…

    1) one advantage of the pipes2py approach is that is gets the output of a pipe into a py environment natively. What missing is a way (or even just an example) for a compiled pipe to consume a json object that already exists in py, maybe by passing it as an arg into a pipe_ function?

    2) one of the complaints i’ve heard devs make about pipes is that you canlt include arbitrary functions. pipes2py natively emits json like objects; point 1 above suggests how it might make sense to allow a compiled pipe to consume json that already exists in the py envt. So what’s missing?
    a) the ability to chain compiled pipes; if 1 is implemented, this follows naturally, simply pass output of one compiled pipe to input of next;
    b) how might we insert an arbirtrary python function into the middle of a compiled pipe ie insert additional functionality within a pipe?

    On a completely separate tangent, I get the feeling that jquery operates on a pipeline model, How far is the approach you took with pipes2py from something along the lines of jquery_pipes?

  5. Tony Hirst

    pipe2py and Google App Engine

    Seems to me that it would be handy to be able to:

    1) host a version of pipe2py and allow users to pass a Yahoo Pipes ID to it and get the compiled code back. If this was done on Google APp engine, could maybe also let users create their own directory of compiled code clips?

    2) provide and ability to generate a python version of a pipe on Google App Engine using pipe2py /and then run that code on Google App Engine. This means Yahoo Pipes could be used as a GUI, and a compiled version generated and then run on Google App Engine (and/or the code exported and run on an arbitrary server as pipe2py code.

    If anyone out there would like to help work up any of these ideas (I have absolutely no experience myself of developing for Google App Engine – it’s still on my to do list;-), please post a reply :-)

  6. Pingback: links for 2010-10-03 | Metamedia
  7. Pingback: It’s All About Flow… « OUseful.Info, the blog…
  8. Pingback: hhhhhhh… « Bibliothèques [reloaded]
  9. Pingback: 5 Minute Hack – QR Codes from an RSS Feed « OUseful.Info, the blog…
  10. Pingback: Backup and Run Yahoo Pipes Pipework on Google App Engine « OUseful.Info, the blog…
  11. Pingback: Yahoo Pipe et Google App Engine : si Yahoo Pipes disparaît, que deviendrons-nous ? « Bibliothèques [reloaded]
  12. klemo

    Hi Tony!

    Great discussion on Pipes alternatives! Recently I’ve been playing with feed parsing in Erlang and eventually I developed small mashup engine for feed filtering. Basically, I prototyped back-end logic for feed remixing end web front-end with small query language for pipes descriptions. More here http://klemo.posterous.com/erl-metafeed-feed-mashup-engine. Project is still just a prototype, but I am planning deployment on public server for testing soon.

    • Tony Hirst

      Looks interesting… Erlang still not something I’ve played with, but for some time I’ve been wondering what sorts of things might be possible if Yahoo Pipes supported pubsubhubbub, webhooks, etc etc, and could play a full role in realtime feed based programming… Let me know when you have an app available for testing:-)

  13. Pingback: On flickr, delicious and Yahoo Pipes… « OUseful.Info, the blog…
  14. Pingback: Just in Case – Saving Your Yahoo Pipes… « OUseful.Info, the blog…