Merging Several Calendar iCal Feeds With Yahoo Pipes

Following up on Displaying Events from Multiple Google Calendars in a Single Embedded Calendar View, and picking up on a quip Jim Groom made in the post that started this thread (“Patrick suggested Yahoo Pipes!, you ever experiment with this? “), I did have a quick play with pipes, and this is what I found..,

The “Fetch Feed” block is happy to accept iCal feeds, as this iCal Merge pipe demonstrates:

(I grabbed the iCal feeds from pages linked to from the Stanford events page. A websearch for “ical lectures events” should pull up other sources;-)

If you import an iCal feed into a Yahoo pipe, you get an iCal output format option:

You can then render this feed in an online calendar such as 30 boxes: pipes merged iCal feeds in 30 boxes (here’s the 30 boxes config page for that calendar).

(NB it’s worth noting that 30 boxes will let you generate a calendar view that will merge up to 3 iCal feeds anyway.)

Using the Pipe’s output iCal URL to try to add the merged calendar feed to Google Calendar didn’t seem to work, but when I converted the URL to a TinyURL (http://tinyurl.com/67bg2d) and used that as the import URL, it worked fine.

Do this:

then this:

and get this:

(I couldn’t get the Yahoo pipe iCal feed to work in iCal on my Mac, nor could I resyndicate the feed from the Google Calendar. I think the problem is with the way the Pipes output URL is constructed… which could be worked around by relaying/republishing the Pipe iCal feed through something with a nice URL, maybe?)

That okay for you, Reverend? :-)

PS having to add the feeds by hand to the pipe is a pain. So how about if we list a set of iCal feeds in an RSS feed (which could be a shared bookmark feed, built around a common tag), then pull that bookmark feed (such as the feed from a delicious page (e.g. http://delicious.com/psychemedia/ical+feedtest)) into a pipe and use it to identify what iCal feeds to pull into the pipe?

Got that? The Loop block grabs the URL for each iCal feed listed in the input RSS feed, and pulls in the corresponding iCal events. It seems to work okay, too:-) That is, the feed powered iCal merge pipe will aggregate events from all the iCal feed listed in the RSS feed that is pulled into the pipe.

So now the workflow, which could possibly be tidied a little, is this:
– bookmark iCal feed URLs to a common somewhere (this can be as weak as shared tags, which are then used as the basis for aggregation of feed URLs);
– take the feed from that common somewhere and pop it into the feed powered iCal merge pipe.
– get the TinyURL of the iCal output from the pipe, and subscribe to it in Google Calendar, (for a personal calendar view).

Hmm… we still can’t publish the Google Calendar though, because we don’t “own” the calendar dates (the iCal feed does)? But I guess we can still use 30boxes as the display surface, and provide a button to add the calendar to Google Calendar?

OKAY – it seems that when you import the feed, it makes sense to tick the box that says “allow other people to find this calendar”:

… because then you can generate some embed code for the calendar, provide a link for anyone else to see the calendar (like this one), and use the tidied up iCal feed that Google calendar now provides to view the calendar in something like iCal:

PPS To make things a little easier, I tweaked the feed powered pipe so now you can just provide it with an RSS feed that points to one or more iCal feeds:

I also added a block to sort the dates in ascending date order. It’s simple enough to add the feed to iGoogle etc, or as a badge in your blog, using the Yahoo Pipes display helper tools:

Hmm, it would be nice if Pipes also offered a “calendar” output view when it knew there was iCal data around, just like it generates a map for when it sniffs geo-data, and a slideshow view when it detects appropriately addressed media objects? Any chance of that, I wonder?

14 comments

  1. redbaiters

    Oh wow, this is groovy. I now take back all the non-programmistan propaganda I spread about Pipes! This is straight forward, and will serve as a nice hacked solution for aggregating ical feeds for UMW. You really are remarkable, the way you work through these solutions openly and meticulously helps me realize how much of what we do is that very process you narrate beautifully here. After this unbelievably helpful series of posts, you mmay have gotten yourself into an awkward position—because after this kind of response I may be trying it far more often :) Thanks Tony!

  2. Tony Hirst

    @jim It’s a pleasure “working” with you; the whole point about OUseful.info is that some of the stuff is useful to someone…everyone in the OU is “too busy” working on “big projects” to see where any of this could lead. Which makes you pretty much my only “user”. Maybe I need to rename the blog MUseful.Info, given that you do seem to be able to find a use for the some of the craziness I come out with;-)

    (One thing to note on the “risk analysis” side – I’m not sure to what extent it makes sense to use Pipes in a production environment…?! ;-)

  3. Tony Hirst

    PS as another “possibly not safe for production use” caveat, note there is also a dependency on the TinyURL resolver working properly…

  4. Pingback: [Geekeries]:Merging Several Calendar iCal Feeds With Yahoo Pipes « OUseful.Info, the blog… « Lelapin’s Weblog
  5. Pingback: links for 2008-12-07 at dekay.org
  6. Pingback: Monkey Vs Robot - My Weekly Idealist post Brokekid.net
  7. mcbenton

    Frickin’ brilliant. I’m curious about one thing. I get how you aggregated the feeds using Delicious and Pipes. Those steps are relatively straightforward for someone who understands how those services work. What I can’t figure out is what made you think to filter your Pipes feed URL through TinyURL? Maybe I don’t understand what tinyurl does, but that step would never have occurred to me in a million years.

    BTW, I used this to aggregate 10 separate calendars over at http://www.ppmrn.net.

    Thanks!

  8. Tony Hirst

    “What I can’t figure out is what made you think to filter your Pipes feed URL through TinyURL? Maybe I don’t understand what tinyurl does, but that step would never have occurred to me in a million years.”

    I’ve been playing with feed pipeline mashups for quite a time now, and have developed several superstitious beliefs over that time…;-)

    There are a couple of problems that can arise relating to the handling of unescaped URIs passed as arguments to a webservice, particularly when it’s ambiguous as to which URI an argument applies to.

    e.g. in service calls constructed according to the patterns http://example.com/service?url=http://example2.com or http://example.com/service/url/http://example2.com
    bad things can happen if the argument URL is not escaped and the example.com service has not been programmed defensively/to accommodate .
    For example, is “arg1” or “arg2” an example.com or example2.com argument in the following cases?
    http://example.com/service?url=http://example2.com?&arg1=foo&arg2=bar
    http://example.com/service/url/http://example2.com?arg1=foo&arg2=bar

    So sometimes, if you “defensively programme” a pipeline mashup, or notice a particular construction keeps failing, it makes sense to try something like http://example.com/service?url=http://tinyurl.com/foobar23

    Note that escaping gets problematic when you have a long pipeline if things aren’t escaped. For example, how do you read this?
    http://example.com/service?url=http://example2.com?arg=foo&arg2=bar&url2=http://example3.com/service?arg=bar&finalArg=true

    Who “owns” arg2, url2 or finalArg? example.com/service, example2.com/service or example3.com/service?

    In some webservices, it may be that there is a limit to the length of a URI that is being passed to the service, in which case reducing the length of the URL you are passing is sometimes worth a try.

    Remember that I play around the pragmatic, scruffy end of the web, too, trying to just get things working for proof of concept demos, and looking for workarounds for the bits that don’t quite work properly yet;-)

  9. 2thyme

    Hey, nice blog here. I’ve been looking at getting this to work for some time, and the tip about tinyURL is a good one. I think I tried FeedBurner or some-such. Anywho. The hangup with getting this deal to work still boils down to how Yahoo Pipes fails to handle recurrent ical events out of gCal. See this discussion (http://tinyurl.com/bw8p39), and let us all know if you can solve it.

  10. Andi

    Hi,

    I investigated why with pipies generated iCal Feeds doesn’t work with iCal (the Mac OS X Application)

    I found out I has to to with the HTTP headers:

    Orinal header from pipes, which iCal doesn’t accept:

    HTTP/1.1 200 OK
    Date: Fri, 03 Jul 2009 14:18:46 GMT
    P3P: policyref=”http://p3p.yahoo.com/w3c/p3p.xml”, CP=”CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV”
    Content-Disposition: attachment; filename=sotm09_test_2.ical
    Connection: close
    Transfer-Encoding: chunked
    Content-Type: text/calendar;

    Header from a ical file with identical content form my webserver which works:
    (i got this file via wget http://pipes.yahoo.com/pipes/pipe.run?_id=pr_5FtNn3hGh95CjY1g5sQ&_render=ical)

    HTTP/1.1 200 OK
    Date: Fri, 03 Jul 2009 14:20:48 GMT
    Server: httpd mod_perl/2.0.4 Perl/v5.10.0
    Last-Modified: Fri, 03 Jul 2009 14:09:47 GMT
    ETag: “601d4-1162-46dcdb57660c0”
    Accept-Ranges: bytes
    Content-Length: 4450
    Connection: close
    Content-Type: text/calendar

    The differences is the missing Content-Disposition: attachment; filename=sotm09_test_2.ical which confuses iCal.

    To test this conclusion create php file on your webserver with the following content:

    Than add the url of that file to iCal and it works

    I posted my conclusions also at
    http://suggestions.yahoo.com/detail/?prop=Pipes&fid=151976

    • Nicholas Barnard

      Just an FYI for folks who read all the way down: I just wandered across this, and Yahoo Pipes now works with iCal on Lion. It might work with Snow Leopard, I’m not sure..

  11. Pingback: Leeds Jazz » Jazz iCal Feeds
    • Tony Hirst

      @Steve Pipes caches for a time (a few minutes last time a checked – which is admittedly a year or two ago!); your browser/O/S, if pulling RSS data from the Pipe, may cache the feed for a time etc etc