Getting an RSS Feed Out of a Google Custom Search Engine (CSE)

Alan posted me a tweet earlier today asking me to prove my “genius” credentials (heh, heh;-):

As far as I know, Google CSEs don’t offer an RSS output (yet: Google websearch doesn’t either, though rumour has it that it will, soon… so maybe CSEs will open up with opensearch too?)

So here’s a workaround…

If you make a query in a Google CSE – such as the rather wonderful How Do I? instructional video CSE ;-) – you’ll notice in the URL an argument that says &cx=somEGobbleDYGookNumber234sTUfF&cof….

google cse

The characters between cx= and either the end of the URL or an ampersand (&) are the ID of the CSE. In the case of How Do I?, the ID is 009190243792682903990%3Aqppoopa3lxa – almost; the “%3A” is a safe encoding for the web of the character “:”, so the actual CSE ID is 009190243792682903990:qppoopa3lxa. But we can work round that, and work with the encoded CSE ID cut straight from the URL.

Using the Google AJAX search API, you can create a query on any CSE that will return a result using the JSON format (a javascript object that can be loaded into a web page). The Google AJAX search API documentation tells you how: construct a Google AJAX web search query using the root http://ajax.googleapis.com/ajax/services/search/web?v=1.0 and add a few extra arguments to pull in results from a particular CSE: Web Search Specific Arguments.

JSON isn’t RSS, but we can get it into RSS quite easily, using a Yahoo pipe…

Just paste in the ID of a CSE (or the whole results URL), add your query, and subscribe to the results as an RSS feed from the More Options menu:

The pipe works as follows…

First up, create a text box to let a user enter a CSE ID cut and pasted from a CSE results page URL (this should work if you paste in the whole of the URL of the results page from a query made on your CSE):

Then create the search query input box, and along with the CSE ID use it to create a URL that calls the Google AJAX API:

Grab the JSON data feed from the Google AJAX Search API and translate the results so that the pipe will output a valid RSS feed:

And there you have it – an RSS feed for a particular query made on a particular Google CSE can be obtained from the Get as RSS output on the pipe’s More Options menu.

Author: Tony Hirst

I'm a Senior Lecturer at The Open University, with an interest in #opendata policy and practice, as well as general web tinkering...

11 thoughts on “Getting an RSS Feed Out of a Google Custom Search Engine (CSE)”

  1. So just picking up on a thread from a later post (the one you did on my video request) – there are more very useful ‘lego’ pieces you are identifying here; howse about a wiki page identifying some of them and pointing to either a post (or the specific tech itself)?

    So this one could go under “JSON->RSS” which I know I’m always trying to recall; the google docs import table /a trick under “grab all hrefs off a page and feed in machine readable format”

    (Though as I think of it I’m not sure a wiki’s the answer, nor that it’s even the right idea that you should do it – nothing stopping any of us lot either, and indeed probably makes more sense [given the variety of ways] for each to keep track of the ones that tickles their fancy. Hmmm…..)

  2. That is pretty darn *clever*. I must say.

    I think you saved me several frustrating weeks of work trying to figure out how to feedify CSEs. My “programming skills” are crap.

    Simply cloning your pipe – with its elegant code – does about 90% of the work. It’s just that last 10% – I wish you had done everything for me. What can I say? Besides my lack of programming skills – I’m lazy: also – getting 100 results for a feed would be nice. Is it simple? Can num=100 be stuck into any CSE url?

    Despite my weak points I do have good ideas. Perhaps even excellent ones.

    A *genius* would have posted a greasemonkey script to finish that last 10%. The script would automatically link CSEs to your pipe so that

    *** any CSE could be subscribed to – in Firefox
    *** – simply by clicking the rss icon in the
    *** address bar.

    And – of course – a wizard of a genius would make the same “easy as pie to subscribe to a feed” available on Google web searches too.

    I can’t believe somebody hasn’t made the pipe + script already. Human beings are strange.

    Please make an account at user Userscripts.org (aka the site for Greasemonkey) – if you don’t have an account there already – and post your script. I’m looking forward to you getting to Feynman level.

    http://userscripts.org/

    – anonymous

  5. Pingback: Getting Lots of Results Out of a Google Custom Search Engine (CSE) via RSS « OUseful.Info, the blog…

  11. Hi,

    The pipes dont work if upgraded to V2. Since yahoo gonna strikeout V1 soon, any advice on how to make it work with V2?

    Thanks in advance.

    Cheers

Comments are closed.

%d bloggers like this: