Tag: yahoo pipes

Yahoo Pipes Retires…

And so it seem that Yahoo Pipes, a tool I first noted here (February 08, 2007), something I created lots of recipes for (see also on the original, archived OUseful site), ran many a workshop around (and even started exploring a simple recipe book around) is to be retired (end of life annoucement)…

Pipes__Rewire_the_web

It’s not completely unexpected – I stopped using Pipes much at all several years ago, as sites that started making content available via RSS and Atom feeds then started locking it down behind simple authentication, and then OAuth…

I guess I also started to realise that the world I once imagine, as for example in my feed manifesto, We Ignore RSS at OUr Peril, wasn’t going to play out like that…

However, if you still believe in pipe dreams, all is not lost… Several years ago, Greg Gaughan took up the challenge of producing a Python library that could take a Yahoo Pipe JSON definition file and execute the pipe. Looking at the pipe2py project on github just now, it seems the project is still being maintained, so if you’re wondering what to do with your pipes, that may be worth a look…

By the by, the last time I thought Pipes might not be long for this world, I posted a couple of posts that explored how it might be possible to bulk export a set of pipe definitions as well as compiling and running your exported Yahoo Pipes.

Hmmm… thinks… it shouldn’t be too hard to get pipe2py running in a docker container, should it…?

PS I don’t think pipe2py has a graphical front end, but javascript toolkits like jsPlumb look like they may do much of the job. (It would be nice if the Yahoo Pipes team could release the Pipes UI code, of course…;-)

PPS if you you need a simple one step feed re-router, there’s always IFTT. If realtime feed/stream processing apps are more your thing, here are a couple of alternatives that I keep meaning to explore, but never seem to get round to… Node-RED, a node.js thing (from IBM?) for doing internet-of-things based inspired stream (I did intend to play with it once, but I couldn’t even figure out how to stream the data I had in…); and Streamtools (about), from The New York Times R&D Lab, that I think does something similar?

Backup and Run Yahoo Pipes Pipework on Google App Engine

Wouldn’t it be handy if you could use Yahoo Pipes as code free rapid prototyping development environment, then export the code and run it on your own server, or elsewhere in the cloud? Well now you can, using Greg Gaughan’s pipe2py and the Google App Engine “Pipes Engine” app.

As many readers of OUseful.info will know, I’m an advocate of using the visual editor/drag and drop feed-oriented programming application that is Yahoo Pipes. Some time ago, I asked the question What Happens if Yahoo Pipes Dies?, partly in response to concerns raised at many of the Pipes workshops I’ve delivered about the sustainability, as well as the reliability, of the Yahoo Pipes platform.

A major issue was that the “programmes” developed in the Pipes environment could only run in that environment. As I learned from pipes guru @hapdaniel, however, it is possible to export a JSON representation of a pipe and so at least grab some sort of copy of a pipes programme. This led to me doodling some ideas around the idea of a Yahoo Pipes Documentation Project, which would let you essentially export a functional specification of a pipe (I think the code appears to have rotted or otherwise broken on this?:-(

This in turn led naturally to Starting to Think About a Yahoo Pipes Code Generator, whereby we could take a description of a pipe and generate a code equivalent version from it.

Greg Gaughan took up the challenge with Pipe2Py (described here) to produce a pipes compiler capable of generating and running Python equivalents of a Yahoo pipe (not all Pipes blocks are implemented yet, but it works well for simple pipes).

And now Greg has gone a step further, by hosting pipe2py on Google App engine so you can make a working Python backup of a pipe in that environment, and run it: Running Yahoo! Pipes on Google App Engine.

As with pipe2py, it won’t work for every Yahoo pipe (yet!), but you should be okay with simpler pipes. (Support for more blocks is being added all the time, and implementations of currently supported blocks also get an upgrade if, as and when any issues are found with them. If you have a problem, or suggestion for a missing block, add a comment on Greg’s blog;-)

(Looking back over my old related posts, deploying to Google Apps also seems to be supported by Zoho Creator.)

Quite by chance, @paulgeraghty tweeted a link to an old post by @progrium on the topic of “POSS == Public Open Source Services: … or User Powered Self-sustaining Cloud-based Services of Open Source Software”:

How many useful bits of cool plumbing are made and abandoned on the web because people realize there’s no true business case for it? And by business case, I mean make sense to be able to turn a profit or at least enough to pay the people involved. Even as a lifestyle business, it still has to pay for at least one person … which is a lot! But forget abandoned … how much cool tech isn’t even attempted because there is an assumption that in order for it to survive and be worth the effort, there has to be a business? Somebody has to pay for hosting! Alternatively, what if people built cool stuff because it’s just cool? Or useful (but not useful enough to get people to pay — see Twitter)?

Well this is common in open source. A community driven by passion and wanting to build cool/useful stuff. A lot of great things have come from open source. But open source is just that … source. It’s not run. You have to run it. How do you get the equivalent of open source for services? This is a question I’ve been trying to figure out for years. But it’s all coming together now …

Enter POSS

POSS is an extension of open source. You start with some software that provides a service (we’ll just say web service … so it can be a web app or a web API, whatever — it runs “in the cloud”). The code is open source. Anybody can fix bugs or extend it. But there is also a single canonical instance of this source, running as a service in the cloud. Hence the final S … but it’s a public service. Made for public benefit. That’s it. Not profit. Just “to be useful.” Like most open source.

Hmmm….. ;-)

Experimenting with the Form…

One of the challenges I’ve set myself this year is to write some sort of book about Yahoo Pipes. Reading Presentation Zen three or four weeks ago, I started to imagine the form such a book might take. What I aspired to was something uncluttered, something that would contrast with the typical confusion of words and ideas that tend to end up being dumped into OUseful.info; something like an artistic recipe book, perhaps, or an art gallery catalogue; the form should be decomposable, allowing sections to be removed or updated without too many side effects on the rest of the work; and the authoring environment should complement the the publication environment, enforcing constraints of the medium the book would be published into.

In short, something like Powerpoint done well, but for print rather than screen.

It seems (of course!) that Tim O’Reilly had already executed a similar idea in the form of the Twitter Book, as John Naughton pointed out to me a couple of days later.

You can read more about O’Reilly’s take on the philosophy behind this sort of representation in Reinventing the Book in the Age of the Web.

Anyway, I spent a weekend doodling ideas, and then left it a couple of weeks. Now I’m looking at it again, and I’d appreciate your comments on whether this sort of presentation works for you, (and if not, why not?), how it might be improved, how it might be simplified (but remain accessible to a novice) and so on. The numbering scheme used is not related to pages – instead, each “point” I make has a number, and these are referred to from the index (I drew inspiration for this sort of numbering from The Pengin Cookery Book). Comments on the level at which the technical content is presented, and the way in which I have started trying to develop a narrative, will also be appreciated.

I originally thought that the “book” should be printed in an A4 landscape form, but then I started to wonder whether two landscape A4 pages could be combined into a portrait A4 page. The font size is problematic, and the I don’t think the same layout works for the landscape vs. portrait view, at least, not as it currently stands.

Anyway, here are the landscape and portrait versions. I don’t think they work as embedded content, which is a shame, but they weren’t written for that sort of medium, so it’s to be expected.

(If you are reading this in a feed reader, you will probably need to click through to the original post in order to see the embedded documents.)

Please bear in mind, too, that I’m not a designer (this much will be be obvious), but that I do think design could play a large part in making this approach usable.

Please feel free to add your comments below:-)