The Real-Time Web and its Relationship with Discovery and Search

As well as a presenting at Online Information this year, I’m also moderating a session on The realtime web: Discovery vs. Search.

To try and frame what I think this topic means to me, I jotted down a set of questions that I’m not sure I have any answers for that I hope will, at least in part, be covered by one or more of the speakers:

Over the last few weeks, the major search engines all announced realtime search capabilities – but what is ‘realtime search’?
 
To what extent can traditional search engine techniques for determining relevancy and ranking results “just cope” with real time signals? Will realtime search focus on ‘turning up’ real time ‘atomic’ status updates, or will it focus on using realtime information as part of a ranking algortihm applied to more traditional web pages? Will ‘social discovery’ or social amplification factors just provide yet another ranking factor to traditional search engines, or is it more complicated than that?
 
In financial markets, being able to publish price information in as near as realtime as possible is key to success, but may also require dedicated hardware and high speed/low latency networks. How ‘real time’ does the real time search’n’discovery web really need to be? Will speed be a determining factor in which ‘realtime’ search engine people use?
 
For som time, it’s ben possibility to identify trends in search behaviour with a variety of periodicities (eg Trendspotting). To what extent does/could/should the ‘periodic web’ influence the results that search engines return. If certain signals tend to lead particular behaviours (eg a spike in a real timee signal on a Thursday predicts a certain bhaviour on a Friday, or a cheer of “goal” on twitter predicts and spike in electricity generation as everyone goes off to boil the kettle and make a cup of tea), might that affect not only search engine rankings in ‘real predictive time’, but also other instrumented systems (such as AdWord pricing, or even energy supply)?
 
To what extent is the realtime web just ‘high frequency’ noise (low effort to produce, quick to disappear) compared to more substanital and expensive to produce ‘low frequency’ signals such as the steady accretion of long lasting links to a web page over time?
 
The web is the eleephant in the room, and it nevr forgets. Given we can now capture, and potentially store, ever increasing amounts of real time sourced data, are w going to need ‘forgetting algorithms’? If so, what might those algorithms do, and over what timescales might they operate?
 
With the increasing instrumentation of the web we are seeing a rise in live “operations” data on the web via services such as Pachube (as well as monthy, quarterly or annual data dumps, such as those released increasingly by government here in the UK, in the US, in Australia and so on). To what extent, if any, might this real time machine collected data about our environment play a role in supporting the (public) discovery of real time events in near real time. (So for example, road traffic information, travel information, weather warnings, earthquake warnings etc)
 
To what extent might realtime data in one medium influence discovery in another – for example, if a lot of photos tagged in a similar way are uploaded to flickr with a particular location, how might that signal; be used?
 
To what extent do the discovery of events in real time impact on the enterprise. What sort of role, if any, is there for a real time, or real time supported capability within a corporate/enterprise/intranet search engine?
 
In th UK, an increasing amount of Linked Data is being made available from government sources. What role, if any, does real time sourced data have to play in the discovery of, or rasoning across, Linked Data? What risks might there be to Linked Data systems by the inclusion or availability of dynamic data, or data that is contiinually generated in real time.

And who are the speakers in the session?

• Stephen Arnold, President, Arnold Information Technology, USA
• Antonio Gulli, Principal Development Manager, STC Europe, Microsoft
• Conrad Wolfram, Co-Founder, Wolfram Research and Wolfram|Alpha, UK
• Morgan Zimmerman, VP Business Development, Exalead, France

 
If there are any other questions that you think need asking, please add them as comment below…