In Getting an RSS Feed Out of a Google Custom Search Engine (CSE), I described a Yahoo! pipe that can be used as a way of getting a RSS feed out of a Google Custom Search Engine using the Google Ajax Search API.
One of the limitations of the API appears to be that it only returns 8 search results at a time. although these can be paged.
So for example, if you run a normal Google search that returns lots of results, those results are presented over several results pages. If you hover over the links for the different pages, and look at the status bar at the bottom of your browser where the link is displayed, you’ll see that the URL for each page of results is largely the same; the difference comes in the &start= argument in the URI that says which number search result should be at the top of the page; something like this:
- http://www.google.com/search?q=space&ie=utf-8&oe=utf-8&aq=t [original query]
- http://www.google.com/search?hl=en&q=space&start=10&sa=N [page 2]
- http://www.google.com/search?hl=en&q=space&start=20&sa=N [page 3]
The same argument – start can be used to page the CSE results from the AJAX API; which means we can add this in to the URI that calls the Google AJAX Search API within a pipe:
This gives us a quick fix for getting more than 8 results out of a CSE: use the get 8 CSE results starting at a given result pipe to get the first 8 results (counted as results 0..7), then another copy of the pipe to get results 9-16 (counted as 8..15 – i.e. starting at result 8), a second copy of the pipe to get results 17-25, and so on, and then aggregate all the results…
Here’s an example – lots of Google CSE results as RSS pipe:
Notice that each CSE calling pipe is called with the same CSE ID and the same search query, but different start numbers.
This resipe hopefully also gives you the clue that you could use the Union pipe block to merge results from different CSEs (just make sure you use the right CSE ID and the right start values!).