Alan posted me a tweet earlier today asking me to prove my “genius” credentials (heh, heh;-):
As far as I know, Google CSEs don’t offer an RSS output (yet: Google websearch doesn’t either, though rumour has it that it will, soon… so maybe CSEs will open up with opensearch too?)
So here’s a workaround…
If you make a query in a Google CSE – such as the rather wonderful How Do I? instructional video CSE ;-) – you’ll notice in the URL an argument that says &cx=somEGobbleDYGookNumber234sTUfF&cof….
The characters between cx= and either the end of the URL or an ampersand (&) are the ID of the CSE. In the case of How Do I?, the ID is 009190243792682903990%3Aqppoopa3lxa – almost; the “%3A” is a safe encoding for the web of the character “:”, so the actual CSE ID is 009190243792682903990:qppoopa3lxa. But we can work round that, and work with the encoded CSE ID cut straight from the URL.
JSON isn’t RSS, but we can get it into RSS quite easily, using a Yahoo pipe…
Just paste in the ID of a CSE (or the whole results URL), add your query, and subscribe to the results as an RSS feed from the More Options menu:
The pipe works as follows…
First up, create a text box to let a user enter a CSE ID cut and pasted from a CSE results page URL (this should work if you paste in the whole of the URL of the results page from a query made on your CSE):
Then create the search query input box, and along with the CSE ID use it to create a URL that calls the Google AJAX API:
Grab the JSON data feed from the Google AJAX Search API and translate the results so that the pipe will output a valid RSS feed:
And there you have it – an RSS feed for a particular query made on a particular Google CSE can be obtained from the Get as RSS output on the pipe’s More Options menu.
11 thoughts on “Getting an RSS Feed Out of a Google Custom Search Engine (CSE)”
So just picking up on a thread from a later post (the one you did on my video request) – there are more very useful ‘lego’ pieces you are identifying here; howse about a wiki page identifying some of them and pointing to either a post (or the specific tech itself)?
So this one could go under “JSON->RSS” which I know I’m always trying to recall; the google docs import table /a trick under “grab all hrefs off a page and feed in machine readable format”
(Though as I think of it I’m not sure a wiki’s the answer, nor that it’s even the right idea that you should do it – nothing stopping any of us lot either, and indeed probably makes more sense [given the variety of ways] for each to keep track of the ones that tickles their fancy. Hmmm…..)
That is pretty darn *clever*. I must say.
I think you saved me several frustrating weeks of work trying to figure out how to feedify CSEs. My “programming skills” are crap.
Simply cloning your pipe – with its elegant code – does about 90% of the work. It’s just that last 10% – I wish you had done everything for me. What can I say? Besides my lack of programming skills – I’m lazy: also – getting 100 results for a feed would be nice. Is it simple? Can num=100 be stuck into any CSE url?
Despite my weak points I do have good ideas. Perhaps even excellent ones.
A *genius* would have posted a greasemonkey script to finish that last 10%. The script would automatically link CSEs to your pipe so that
*** any CSE could be subscribed to – in Firefox
*** – simply by clicking the rss icon in the
*** address bar.
And – of course – a wizard of a genius would make the same “easy as pie to subscribe to a feed” available on Google web searches too.
I can’t believe somebody hasn’t made the pipe + script already. Human beings are strange.
Please make an account at user Userscripts.org (aka the site for Greasemonkey) – if you don’t have an account there already – and post your script. I’m looking forward to you getting to Feynman level.
This has solved a fairly major issue for me.
So, thank you!
How do we get more than 8 results?
Today I got another (email) request for a pipe that will produce more than 8 results from a CSE; I’ve just posted a recipe for how to do that in the OUseful.info post “Getting Lots of Results Out of a Google Custom Search Engine (CSE) via RSS” [ https://ouseful.wordpress.com/2008/12/17/getting-lots-of-results-out-of-a-google-custom-search-engine-cse-via-rss/ ]
Genius! sheer genius!
Thanks for the walk through – this is exactly what I was looking for. Very useful.
Hi, I was planning the same type with this, but is it legal for direct linking the keyword to the custom search?
this is the sample for my yahoo pipes trends custom search…
let me know what do you think about it…
The pipes dont work if upgraded to V2. Since yahoo gonna strikeout V1 soon, any advice on how to make it work with V2?
Thanks in advance.
Comments are closed.