Libraries Near Me Map (Courtesy of LibraryThing)

Given your location as postcode, where can you find a list of libraries near you? As with many of these public information style questions, it can be quite hard trying to find a single general answer. A crude way is to enter a search map along the lines of libraries near POSTCODE into Google Maps, but the results that are returned aren’t necessarily that good… Alternatively, you can go to your local council website, and then do a search there for libraries; but the format in which results are provided can vary; and if you’re a developer, there’s no immediately obvious way of creating (or consuming) a service that will allow you to create a small embeddable map showing the location of libraries in the vicinity of a particular postcode area.

Putting a tweet out last week to ask if anyone knew of a “library lookup by postcode”, several people suggested Worldcat, but @lynncorrigan suggested LibraryThing Local. Hmmm… here’s what it offers:

That is, we can search by postcode, limit results to book related venues within a particular distance, and then filter down by library type, getting both a list of results back as well as map view. On running a search, we also get a URL that contains the search term as a parameter:

http://www.librarything.com/local/place/mk7%206aa%2C%20uk

This was a good start, so I had a look around the LibraryThing APIs to see if an API was available for the local service. I couldn’t spot anything, so I sent @librarythingtim a tweet to check, can got a confirmation back that there was no API for that service… Hmmm…

At times like this, it’s often worth trying different searches, and clicking different links (such as the “All”, “Bookstore” and “Library” fitlers, as well as the distance setting), to see what happens to the page – whether the URL changes for instance. It’s also worth doing a View Source on the page to see if there are any immediately obvious calls to AJAX web services that bring content into the page on those times when a setting change doesn’t cause the page to reload (with or without the same URL), but you suspect that there may have been a call to a webservice somewhere.

In the case of LibraryThing Local, clicking the links and changing the settings changed the displayed results without appearing to reload the page, but I wasn’t sure whether the change was just based on data stored within the web page, or whether the information was being pulled in from somewhere else.

Which is where developer tools can come in useful… One tool I use every now and gain is a Firefox extension called Firebug. This debugging environment operates within Firefox, and among other things can be used to track calls that a web page makes back to a server, along with any arguments passed to the server, and any results returned from it. Clicking between the “All” and “Libraries” links suggested that corresponding data was being pulled in to the page from a LibraryThing server:

The call was being made to http://www.librarything.com/ajax_venuesNearUser.php and returned HTML data that could be displayed within the results listing:

Hmmm…

The next step was to see if I could construct a URL without and session identifying variables to pull similar data back, such as:
http://www.librarything.com/ajax_venuesNearUser.php?&q=mk76aa,uk&showvenue=2

This did indeed work. The HTTP call also suggested there was a parameter identified as d. Could this be for distance? I tried it, setting it to 2, then 3. It seemed to work:-)

The next step was to try to pull the results into some sort of environment where I could play with it. The original motivation for getting this data was so that I could add libraries to a map of UK Online centres, based on my 5 Minute Hack – UK Centres Online Map, which used Yahoo pipes to geocode UK Online Centre addresses and generate a KML feed listing them in the vicinity of a particular postcode… So Yahoo pipes it was to be…

As well as RSS feeds, Yahoo Pipes can ingest XML and JSON data, as well as HTML pages. My first thought was to try to import the LibraryThing Local data as XML… but for some reason the Pipes Fetch Data block failed to parse it. Trying the Fetch Page block also failed to work… Hmmm…

At this point, I cast my mind around for other tools that might do the job. Google Spreadsheets? No; if the library data was in a table or a list, we might be able to use the importHTML formula, but is isn’t… Maybe I could try to import the HTML as XML…? Hmmm… XML… How about I try using YQL to parse the :LibraryThing Local data, and then pull the result into the pipe?

Success:-)

Here’s the query I used:
select * from html where
url=”http://www.librarything.com/ajax_venuesNearUser.php?showvenue=2&q=MK7+6AA&d=3″ and xpath=’//div[@class=”venueItem”]’

Rather conveniently, there is a YQL block available in Yahoo Pipes that can be provided with a YQL query and will pull the result back into the Pipes environment. Now the pipework can begin… but what do we want it to do?

The recipe will follow similar lines to the UK Online centres map pipe: grab a postcode from the user (and the search radius for good measure); construct the LibraryThing Local URL to fetch the raw data; use YQL to get the data into the pipes environment, then identify the address of each library, geocode it, create a title and piece of descriptive text (if required) then output the who lot. Simple:-)

To start then, let’s construct the LibraryThing Local URL:

The reason we use a Create URL block is so that the postcode is automatically escaped (e.g. if the user places space characters in the postcode when they enter it, these need handling appropriately).

Having got the URL, we need to insert it into a YQL query – I use a bit of string replacing magic for that;-)

Here’s the base part of the YQL query:
select * from html where
url=”MAGIC” and xpath=’//div[@class=”venueItem”]’

So, now we’re in a position to grab the local library data from LibraryThing Local via YQL:

In the results, we see several separate items, one for each Library, but within each item, there are two “subresults”. Fortunately, these are “regular”, in that the result(s) for each Library is structured in the same way. Knowing how to handle the subresults required a little bit of Pipes trickery:

It’s worth just looking at what’s going on in that block for a moment, and comparing it to the results that were coming from the YQL processor…

All that remains to do now is geocode the address of each library, and output the result:

Here’s the result:

You can see the pipe here: http://pipes.yahoo.com/ouseful/librarylocations

For details on how to use the output of this pipe to create your own embeddable maps, see the second half of 5 Minute Hack – UK Centres Online Map, in particular how to take the KML output of the pipe and use it to display the results in an embeddable Google map.

PS via @ostephens libraries lookup on People’s Network: “you can specify postcode and radius & get results as XML or route to xslt”