Will Lack of Relevancy be the Downfall of Google?

Every so often, posts come around about new search engines that are going to make a bid to become a Google search killer, but I wonder if the changing nature of the web itself will lead people to a search engine that appears to do search better in those bits of the web that they’re spending most time, and so lead them away from Google?

It’s hard thinking back the 10 years or so to a time before Google, so I’m not sure what prompted me to switch allegiance from Metacrawler to Google? Maybe it was that Google results were dominating the Metacrawler results page? (For those of you who have know idea what I’m talking about, Metacrawler (which lives on to this day, as… MetaCrawler;-) was essentially a federated search engine, that pooled results from several, early web search engines. Before Metacrawler, I used Webcrawler, which was one of the first search engines to do full text search, I think?

In those early days, Google won out on producing “better” results in part because of its PageRank algorithm, in part becuase of its speedy response. PageRank essentially determines the authority of a page by the number of pages that link to it, and the authority of those pages. There’s lots of other voodoo magic in the ranking and relevancy algorithm now, of course, but that was at the heart of what made Google different in the early days.

So Google came good in large part because it used the structure of the web to help people better navigate the web.

But what of the structure of the web now? Many of the recently launched search engines have made great play of being “social” or “people powered” search engines, that leverage personal recommendations to improve search results. The big search engines are experimenting with tools that let searchers “vote up” more relevant results, and so on (e.g. Google’s experiment, or Microsoft’s URank experiment).

But it might be that the nature of recommending a page to someone else is now less to do with publishing a link to another site on a web page or in a blog post, than sharing a link with someone in a more conversational way (though as to how you found that link in the first place – there lies a problem;-)

So although Google won’t be able to snoop on link sharing in “walled garden” social networks like Facebook, I wonder if they are tracking link sharing in services like Twitter? (Google owns rival microblogging site Jaiku, but since buying it, all has been quiet. Maybe they’re waiting for the masses to become conscious of the thing called Twitter, then they’ll go prime time with Jaiku?)

Just by the by, there’s also the “problem” that many shared links are now being obfuscated by URL shortening services, which means that TinyURLs, bit.ly URLs and is.gd URLs all need resolving back to the pages they point to in order to rank those pages. (Hmm…. so when will the Goog be pushing it’s own URL shortening service, I wonder?)

This link resolution is easy enough to achieve, though. For example, the Tiwtturly service tracks the most poplular links being shared on Twitter over a 24 hour period (I think they also used to let you see who was tweeting about a particular URL, because I built a pipe around it – A Pipe for Twitturly – although that functionality appears to have disappeared?)

PS Maybe that Jaiku launch moment will actually be on mobile devices – on the iPhone (which now has Google Voice search), and on Android devices? Maybe Jaiku’s relaunch (and remember, Jaiku was heavy on the mobile stuff) will be a defining moment that hails: “the era of the PC [i]s over,… the future belong[s] to cloud applications accessed via phones”, via Daddy, Where’s Your Phone? which also includes a lovely story to illustrate this: a child overhears her dad answering “I don’t know” to a question

“Daddy, where’s your phone?”

“What do you mean, where’s my phone?” She explained that she’d overheard the question. Why wasn’t he just looking up the answer on his phone?

Cf. the apocryphal story of the child looking behind the TV set for a mouse, and the idea that “a screen without a mouse is broken”… ;-)

Recession, What Recession?

Following on from my own Playing With Google Search Data Trends, and John’s Google’s predictive power (contd.) pick up on this post from Bill Thompson, The net reveals the ties that bind, here’s one possible quick look at the impending state of the recession…

What search terms would you say are recession indicators?

PS I wonder to what extent, if any, the financial wizards factor real time “search intent” tracking into their stock trading strategies?

PPS I have to admit, I don’t really understand the shape of this trend at all?

The minima around Christmas look much of a muchness, but the New Year pear – and then the yearly average, are increasing, year on year? Any ideas?

The Future of Search Is Already Here

One of my favourite quotes (and one I probably misquote – which is a pre-requisite of the best quotes) is William Gibson’s “the future is already here, it’s just not evenly distributed yet”…

Several times tonight, I realised that the future is increasingly happening around me, and it’s appearing so quickly I’m having problems even imagining what might come next.

So here for you delectation are some of the things I saw earlier this evening:

  • SnapTell: a mobile and iPhone app that lets you photograph a book, CD or game cover and it’ll recognise it, tell you what it is and take you to the appropriate Amazon page so you can buy it… (originally via CogDogBlog;

  • Shazam, a music recognition application that will identify a piece of music that’s playing out loud, pop up some details, and then let you buy it on iTunes or view a version of the song being played on Youtube (the CogDog also mentioned this, but it was arrived at tonight independently);

    So just imagine the “workflow” here: you hear a song playing, fire up the Shazam app, it recognises the song, then you can watch someone play a version of the song (maybe even the same version on Youtube.

  • A picture of a thousand words?: if you upload a scanned document onto the web as a PDF document, Google will now have a go at running an OCR service over the document, extracting the text, indexing it and making it searchable. Which means you can just scan and post, flag the content to the Googlebot via a sitemap, and then search into the OCR’d content; (I’m not sure if the OCR service is built on top of the Tesseract OCR code?)
  • barely three months ago, Youtube added the ability to augment videos with captions. With a little bit of glue, the Google translate service will take those captions and translate them into another language for you (Auto Translate Now Available For Videos With Captions):

    “To get a translation for your preferred language, move the mouse over the bottom-right arrow, and then over the small triangle next to the CC (or subtitle) icon, to see the captions menu. Click on the “Translate…” button and then you will be given a choice of many different languages.” [Youtube blog]

Another (mis)quote, this time from Arthur C. Clarke: “any sufficiently advanced technology is indistinguishable from magic”. And by magic, I guess one thing we mean is that there is no “obvious” causal relationship between the casting of a spell and the effect? And a second thing is that if we believe something to be possible, then it probably is possible.

I think I’m starting to believe in magic…

PS Google finally got round to making their alerts service feed a feed: Feed me! Google Alerts not just for email anymore, so now you can subscribe to an alerts RSS feed, rather than having to receive alerts via email. If you want to receive the updates via Twitter, just paste the feed URL into a service like Twitterfeed or f33d.in.

PPS I guess I should have listed this in the list above – news that Google has (at least in the US) found a way of opening up its book search data: Google pays small change to open every book in the world. Here’s the blog announcement: New chapter for Google Book Search: “With this agreement, in-copyright, out-of-print books will now be available for readers in the U.S. to search, preview and buy online — something that was simply unavailable to date. Most of these books are difficult, if not impossible, to find.”

Time to Get Scared, People?

Last week, I posted a couple of tweets (via http://twitter.com/psychemedia) that were essentially doodles around the edge of what services like Google can work out about you from your online activity.

As ever in these matters, I picked on AJCann in the tweets, partly because he evangelises social web tool use to his students;-)

So what did I look at?

  • the Google Social Graph API – a service that tries to mine your social connections from public ‘friendships’ on the web. Check out the demo services…

    For example, here’s what the Google social API can find from Alan’s Friendfeed account using the “My Connections” demo:

    • people he links to on twitter and flickr;
    • people who link to him as a contact on twitter, delicious, friendfeed and flickr;
    • a link picked up from Science of the Invisible (which happens to be one of Alan’s blogs), also picks out his identi.ca identity; adding that URL to the Social Graph API form pulls out more contacts – via foaf records – from Alan’s identi.ca profile;

    The “Site Connections” demo pulls out all sorts of info about an individual by looking at URLs prominently associated with them, such as a personal blog:

    The possible connections reveal Alan’s possible identity on Technorati, Twitter, identi.ca, friendfeed, swurl, seesmic and mybloglog.

  • For anyone who doesn’t know what Alan looks like, you can always do a “face” search on Google images;
  • increasingly, there are “people” search engines out there that are built solely for searching for people. One example is Spock (other examples include pipl, zoominfo and wink [and 123people, which offers and interesting federated search results page]). The Spock “deep web” search turns up links that potentially point to Alan’s friendfeed and twitter pages, his revver videos, slideshare account and so on;
  • Alan seems to be pretty consistent in the username he uses on different sites. This makes it easy to guess his account on different sites, of course – or use a service like User Name Check to do a quick search;

Now I wasn’t going to post anything about this, but today I saw the following on Google Blogoscoped: Search Google Profiles, which describes a new Google search feature. (Didn’t know you had a Google Profile? If you have a Google account, you probably do – http://www.google.com/s2/profiles/me/? And if you want to really scare yourself with what your Google account can do to you, check http://www.google.com/history/… go on, I dare you…)

I had a quick look to see if I could find a link for the new profile search on my profile page, but didn’t spot one, although it’s easy enough to find the search form here: http://www.google.com/s2/profiles. (Maybe I don’t get a link because my profile isn’t public?)

Anyway, while looking over my profile, I thought I’d add my blog URL (http://ouseful.info) to it – and as soon as I clicked enter, got this:

A set of links that I might want to add to my profile – taken in part from the Social Graph API, maybe? Over the next 6 months I could see Google providing a de facto social network aggregation site, just from re-posting to you what they know about your social connections from mining the data they’ve crawled, and linking some of it together…

And given that the Goog can learn a lot about you by virtue of crawling public pages that are already out there, how much more comprehensive will your profile on Google be (and how certain will it be in the profile it can automatically generate around you?) if you actually feed it yourself? (Bear in mind things like health care records exist already…)

PS I just had a look at my own Web History page on Google, and it seems like they’ve recently added some new features, such as “popular searches related to my searches”, and also something on search trends that I don’t fully (or even partially) understand? Or maybe they were already there and I’ve not noticed before/forgotten (I rarely look at my search history…)

PPS does the web know when your birthday is??? Bewar of “Happy Birthday me…”. See also My Web Birthday.

[Have you heard about Google’s ‘social circle’ technology yet? read more]

Getting an RSS Feed Out of a Google Custom Search Engine (CSE)

Alan posted me a tweet earlier today asking me to prove my “genius” credentials (heh, heh;-):

As far as I know, Google CSEs don’t offer an RSS output (yet: Google websearch doesn’t either, though rumour has it that it will, soon… so maybe CSEs will open up with opensearch too?)

So here’s a workaround…

If you make a query in a Google CSE – such as the rather wonderful How Do I? instructional video CSE ;-) – you’ll notice in the URL an argument that says &cx=somEGobbleDYGookNumber234sTUfF&cof….

google cse

The characters between cx= and either the end of the URL or an ampersand (&) are the ID of the CSE. In the case of How Do I?, the ID is 009190243792682903990%3Aqppoopa3lxa – almost; the “%3A” is a safe encoding for the web of the character “:”, so the actual CSE ID is 009190243792682903990:qppoopa3lxa. But we can work round that, and work with the encoded CSE ID cut straight from the URL.

Using the Google AJAX search API, you can create a query on any CSE that will return a result using the JSON format (a javascript object that can be loaded into a web page). The Google AJAX search API documentation tells you how: construct a Google AJAX web search query using the root http://ajax.googleapis.com/ajax/services/search/web?v=1.0 and add a few extra arguments to pull in results from a particular CSE: Web Search Specific Arguments.

JSON isn’t RSS, but we can get it into RSS quite easily, using a Yahoo pipe…

Just paste in the ID of a CSE (or the whole results URL), add your query, and subscribe to the results as an RSS feed from the More Options menu:

The pipe works as follows…

First up, create a text box to let a user enter a CSE ID cut and pasted from a CSE results page URL (this should work if you paste in the whole of the URL of the results page from a query made on your CSE):

Then create the search query input box, and along with the CSE ID use it to create a URL that calls the Google AJAX API:

Grab the JSON data feed from the Google AJAX Search API and translate the results so that the pipe will output a valid RSS feed:

And there you have it – an RSS feed for a particular query made on a particular Google CSE can be obtained from the Get as RSS output on the pipe’s More Options menu.

Google Personal Custom Search Engines?

A couple of days ago, I gave a talk about possible future library services, and in passing mentioned the way that my Google search results are increasingly personalised. Martin picked up on this in a conversation over coffee, and then in a blog post (“Your search is valuable to us“):

This made me think that your search history is actually valuable, because the results you get back are a product of the hours you have invested in previous searches and the subject expertise in utilising search terms. So, if you are an expert in let’s say, Alaskan oil fields, and have been researching this area for years, then the Google results you get back for a search on possible new oil fields will be much more valuable than the results anyone else would get.

[I]f you can assemble and utilise the expert search of a network of people, then you can create a socially powered search which is very relevant for learners. Want to know about really niche debates in evolution? We’ve utilised Richard Dawkins, Steve Jones and Matt Ridley’s search history to give you the best results. Or if you prefer, the search is performed as the aggregation of a specialist community.

There are more than a few patents in this area of course (you can get a feel for what the search engines are (thinking about) doing in this area by having a read through these SEO by the SEA posts on “search+history+personal”), but I was wondering:

how easy would it be to expose my personal search history reranking filter (or whatever it is that Google uses) as a custom search engine (under my privacy controls, of course)?

As Martin says (and as we discussed over coffee), you’d want to disable further personalisation of your CSE by users who aren’t you (to get round the Amazon equivalent of Barbie doll and My Little Pony “items for you” recommendations I seem to get after every Christmas!), but exposing the personal search engine would potentially be a way of exposing a valuable commodity.

In the context of the Library, rather going to the Library website and looking up the books by a particular author, or going to ORO and looking up a particular author’s papers, you might pull their personalised search engine off the shelf and use that for a bit of a topic related Googling…

In a comment to Martin’s post, Laura suggests “Aren’t the search results that the expert actually follows up and bookmarks more powerful? Every academic should be publishing the RSS feeds for their social bookmarks, classified by key terms. The user can choose to filter these according to the social rating of the URL and aggregate results from a group of experts according to their reputation in their field and their online expertise in finding valuable sources.”

I guess this amounts to a social variant of the various “deliSearch” search engines out there, that let you run a search over a set of bookmarked pages or domains (see Search Hubs and Custom Search at ILI2007, for example, or these random OUseful posts on delicious powered search etc)?

At times like these, I sometimes wish I’d put a little more effort into searchfeedr (example: searching some of my delicious bookmarks tagged “search’ for items on “personal search”). I stopped working on searchfeedr before the Google CSE really got going, so it’d be possible to build a far more powerful version of it now…

Anyway, that’s maybe something else to put on the “proof-of-concept to do” list…

PS Note to self – also look at “How Do I?” instructional video search engine to see how easy it would be to embed videos in the results…

What Google Thinks of the OU…

More and more search boxes now try to help the user out by making search completion recommendations if you pause awhile when typing query terms into a search box.

So here’s how you get helped out on Youtube:

And here’s what Google suggest is offering on a default (not signed in) Google personal page:

Here’s Yahoo:

Google Insights for Search also provides some food for thought from a free tool you can run against any search terms that get searched on enough. So here for example is the worldwide report for searches on open university over the last 90 days:

Tunneling down to look at searches for open university from the UK, I notice quite a lot were actually looking for information about university open days… Hmmm… do we have a permanent “open day” like web page up onsite anywhere, I wonder?

Let’s see – after all, the OU search engine never fails…

… to provide amusement…

Google comes up with:

Would it make sense, I wonder, to try to capitalise on the name of the university and pull traffic in to a landing page specifically designed to siphon off Google search traffic from students looking for open days at other universities? ;-)

“The Open University: where every day is a university open day. From Newcastle to Bristol, London to Leeds, Oxford to Cambridge, Birmingham to Edinburgh, Cardiff to Nottingham, why not pop in to your local regional Open University center to see what Open University courses might be right for you?”, or somesuch?! Heh heh… :-)

ORO Results in Yahoo SearchMonkey

It’s been a long – and enjoyable – day today (err, yesterday, I forgot to post this last night!), so just a quick placeholder post, that I’ll maybe elaborate on with techie details at a later date, to show one way of making some use of the metadata that appears in the ORO/eprints resource splash pages (as described in ORO Goes Naked With New ePrints Server): a Yahoo SearchMonkey ORO augmented search result – ORO Reference Details (OUseful).

The SearchMonkey extension – which when “installed” in your Yahoo profile, will augment ORO results in organic Yahoo search listings with details about the publication the reference appears in, the full title (or at least, the first few characters of the title!), the keyowrds used to describe the reference and the first author, along with links to a BibTeX reference and the document download (I guess I could also add a link in there to a full HTML reference?)

The SearchMonkey script comes in two parts – a “service” that scrapes the page linked to from the results listing:

And a “presentation” part, that draws on the service to augment the results:

It’s late – I’m tired – so no more for now; if you interested, check out the Yahoo SearchMonkey documentation, or Build your own SearchMonkey app.

Special Interest Custom Search Engines

A recent post by Downes (PubMed Now Indexes Videos of Experiments and Protocols in Life Sciences) reminded me of a Google custom search engine I started to put together almost a year or ago to provide a meta-search over science experiment protocols.

At the time, I managed to track three likely sites down, although despite my best intentions when I created the initial CSE, I haven’t managed even cursory maintenance of the site.

Anyway, for what it’s worth, here’s a link to my Science Experimental Protocols Video Search (a search for DNA will show you what sorts of results are typical). If you know of any other sites that publish scientific experimental protocols, please fee free to post a link in the comments to the post.

Another custom search engine I started looking at at the start of this year, inspired by a conversation with a solicitor friend over New Year, was a search of UK (English and Scottish) legislation. The intention here was to come up with a CSE that could provide a value adding vertical search site to a legal website. If i remember correctly (?!;-) the CSE only took an hour or so pull together, so even though we never pursued embedding it on live website, it wasn’t really that much time to take out…

If you want to check it out, you can find it here: LegalDemo.

One CSE I do maintain is “How Do I?”, a metasearch engine over instructional video websites. There are almost as many aggregating websites of this ilk as there are sites publishing original instructional content, but again, it didn’t take long to pull together, and it’s easy enough to maintain. You can find the search engine here: “How Do I?” instructional video metasearch engine, and a brief description of its origins here: “How Do I…” – Instructional Video Search.

Another 10 minute CSE I created, this time following a comment over a pint about the “official” OpenLearn search engine, was an OpenLearn Demo CSE (as described here: OpenLearn Custom Search).

And finally (and ignoring other the other half-baked CSEs I occasionally dabble with), there’s the CSE I’ve been doodling with most recently: the OUseful search engine (I need to get that sorted on a better URL..). This CSE searches over the various blogs I’ve written in the past, and write on at the moment. If you want to search over posts from the original incarnation of OUseful.info, this is one place to do it…

Just looking back over the above CSEs, I wonder again about who’s job it is (if anyone’s), to pull together and maintain vertical search engines in an academic environment, or show students how they can crate their own custom search engines? (And one level down from that, who’s role is it to lead the teaching of the “search query formulation” information skill?)

In the OU at least, the Library info skills unit have been instrumental in engaging with course teams to develop information literacy skills, as well as leading the roll out of Beyond Google… but I have to admit, I do wonder just how well equipped they are to helping users create linked data queries, SPARQL queries, or SQL database queries containing a handful of joins? (I also wonder where we’re teaching people how to create pivot tables, and the benefits of them…?!)

Thinking about advanced queries, and the sighs that go up when we talk about how difficult it is to persuade searchers to use more than two or three keyword search terms, I’ve also been wondering what the next step in query complexity is likely to be after the advanced search query. And it strikes me that the linked data query is possibly that next step?

Having introduced the Parallax Freebase interface to several people over the last week, it struck me that actually getting the most out of that sort of interface (even were Freebase populated enough for more than a tiny minority of linked queries to actually work together) is not likely to be the easiest of jobs, particularly when you bear in mind that it’s only a minority of people who know how to even conceptualise advanced search queries, let alone know how to construct them at a syntactic level, or even via a web form.

The flip side to helping users create queries is of course helping make information amenable to discovery by search, as Lorcan Dempsey picks up on in SEO is part of our business. Here again we have maybe another emerging role for …. I don’t know…? The library? And if not the library, then whom?

(See also: The Library Flip, where I idly wondered whether the academic library of the future-now should act “so as to raise the profile of information it would traditionally have served, within the search engine listings and at locations where the users actually are. In an academic setting, this might even take the form of helping to enhance the reputation of the IP produced by the institution and make it discoverable by third parties using public web search engines, which in turn would make it easy for our students to discover OU Library sponsored resources using those very same tools.”)

PS Just a quick reminder that there are several OU Library job vacancies open at the moment. You can check them out here: OU Library Jobs Round-Up (August 2008).

Searching for Curriculum Development Course Insights

For almost as long as I can remember (?! e.g. Search Powered Predictions), I’ve had the gut feeling that one of the most useful indicators about the courses our students want to study is their search behaviour, both in terms of searches that drive (potential) students to the OU courses and qualifications website from organic search listings, as well as their search behaviour whilst on the OU site, and whilst floundering around within the courses and quals minisite.

A quick skim through our current strategic priorities doc (OU Futures 2008 (internal only), though you can get a flavour from the public site: Open University Strategic Priorities 2007) suggests that there is increased interest in making use of data, for example as demonstrated by the intention to develop a more systematic approach for new curriculum developments, such that the student market, demography and employment sectors are the primary considerations.

So, to give myself something to think about over the next few days/weeks, here’s a marker post about what a “course search insights” tool might offer, inspired in part by the Google Youtube Insights interface.

So, using Youtube Insight as a starting point, let’s see how far we can get…

First off, the atom is not a Youtube video, it’s a course, or to be more exact, a course page on the courses and quals website… Like this page for T320 Ebusiness technologies: foundations and practice for example. The ideas are these: what might an “Insight” report look like for a course page such as this, how might it be used to improve the discoverability of the page (and improve appropriate registration conversion rates), and how might search behaviour inform curriculum development?

Firstly, it might be handy to segment the audience reports into four:

  • people hitting the page from an organic search listing;
  • people hitting the page from an internal (OU search engine) search listing;
  • people hitting the page from an ‘organic’ link on a third party site (e.g. a link to the course page from someone’s blog);
  • people hitting the page from an external campaign/adword etc on a search engine;
  • people hitting the page from any other campaign (banner ads etc);
  • the rest…

For the purposes of this post, I’ll just focus on the first two, search related, referrers… (and maybe the third – ‘organic’ external links). What would be good to know, and how might it be useful?

First off, a summary report of the most popular search terms would be handy:

– The terms used in referrers coming from external organic search results give us some insight into the way that the search engines see the page – and may provide clues relating to how to optimise the page so as to ensure we’re getting the traffic we expect from the search engines.

– The terms used within the open.ac.uk search domain presumably come from (potential) students who have gone through at least one micro-conversion, in that they have reached, and stayed in, the OU domain. Given that we can (sometimes) identify whether users are current students (e.g. they may be logged in to the OU domain as a student) or new to the OU, there’s a possibility of segmenting here between the search terms used to find a page by current students, and new prospects.

(Just by the by, I emailed a load of OU course team chairs a month or two ago about what search terms they would expect potential students to use on Google (or on the OU search engine) to find their course page on the courses and quals site. I received exactly zero responses…)

The organic/third party incoming link traffic can also provide useful insight as to how courses are regarded from the insight – an analysis of link text, and maybe keyword analysis of the page containing the link – can provide us with clues about how other people are describing our courses (something which also feeds into the way that the search engines will rank our course pages; inlink/backlink analysis can further extend this approach.). I’m guessing there’s not a lot of backlinking out there yet (except maybe from professional societies?), but if and when we get an affiliate scheme going, this may be one to watch…?

So that’s one batch of stuff we can look at – search terms. What else?

As a distance learning organisation, the OU has a national reach (and strategically, international aspirations), so a course insight tool might also provide useful intelligence about the geographical location of users looking at a particular course. Above average numbers of people reading about a course from a particular geo-locale might provide evidence about the effectiveness of a local campaign, or even identify a local need for a particular course (such as the opening or closure of large employer).

The Youtube Insight reports shows how as the Google monster gets bigger, it knows more and more about us (I’m thinking of the Youtube Insight age demographic/gender report here). So providing insight about the gender split and age range of people viewing a course may be useful (we can find this information out for registered users – incoming users are rather harder to pin down…), and may provide further insight when these figures are compared to the demographics of people actually taking the course, particularly if the demographic of people who view a course on the course catalogue page differs markedly from the demographics of people who take the course…

(Notwithstanding the desire to be an “open” institution, I do sometimes wonder whether we should actually try to pitch different courses at particular demographics, but I’m probably not allowed to say things like that…;-)

As well as looking at search results that (appear) to provide satisfactory hits, it’s also worth looking at the internal searches that don’t get highly relevant results. These searches might indicate weak optimisation of pages – appropriate search terms donlt find appropriate course pages – or they might identify topics or courses that users are looking for that don’t exist in the current OU offerings. Once again, it’s probably worth segmenting these unfulfilled/unsatisfactory courses according to new prospects and current students (and maybe even going further, e.g. by trying to identify the intentions of current students by correlating their course history with their search behaviour, we may gain insight into emerging preferences relating to free choice courses within particular degree programmes).

To sum up… Search data is free, and may provide a degree of ‘at arms length’ insight about potential students before we know anything about them ‘officially’ by virtue of them registering with us, as well as insight relating to emerging interests that might help drive curriculum innovation. By looking at data analysis and insight tools that are already out there, we can start to dream about what course insight tools might look like, that can be used to mine the wealth of free search data that we can collect on a daily basis, and turn it into useful information that can help improve course discovery and conversion, and feed into curriculum development.