Free the slaves from the webmasters

Bradley Horowitz comes close to bashfulness over the freedom he enjoys at Yahoo: “It’s almost too much. They [chief executive Terry Semel and his team] are giving me all the leash in the world. They are just pouring gasoline on the fire and telling me to go for it,” he grins, mixing metaphors enthusiastically.

Sunnyvale, California-based Yahoo, now more than 10 years old, virtually invented the technology for finding information on the web but, in recent years, it has seen its star eclipsed by the upstart Google, contributor of the verb “to Google” to the English language, and by an increasingly aggressive Microsoft. Now it is fighting back on multiple fronts, not least of which is its core competence, web search.

Enter Mr Horowitz, an entrepreneurial graduate of the MIT Media Lab. He founded Virage, a media indexing company eventually sold to the UK group Autonomy two years ago, and was appointed Yahoo technology director, search and marketplace group, 18 months ago. It is a ponderous title that ill reflects the speed with which he has to redefine the way society interacts with the web if Yahoo is to regain its former lustre. His group hopes to usher in a new concept of web searching where pure search à la Google or Microsoft’s MSN will be augmented, if not wholly replaced, by the combined expertise and experience of the entire body of net users. Yahoo calls this “social search”: Mr Horowitz claims it will be the defining characteristic of the next generation of web search tools.

In pursuit of this vision of the future, one of his first moves after being appointed was to persuade his new employers to spend some $30m (£17.4m) on Flickr, an online photographic store and share service with a “tagging” technology that makes it easy for subscribers to create a hierarchy of favourite images.

Yahoo has now launched an experimental service based on Flickr technology called MyWeb 2.0, which enables customers to tag their favourite websites and to have access to websites tagged by other people – specifically friends and friends of friends: two degrees of separation, as the jargon has it. Mr Horowitz gives the example of typing “bicycle” into a conventional search engine. It produces millions of results, but the results will be the same for you, for your neighbour and for thousands of people across the world. MyWeb 2.0, on the other hand, will return sites that you have already tagged as valuable together with those selected by friends and acquaintances. It is the equivalent of asking your buddies: “Who knows a good bike shop?”

To Yahoo, it is the fourth chapter in the web search story. The first, initiated by Yahoo’s founders, the Stanford University postgraduates David Filo and Jerry Yang, was a simple directory, a manually prepared classification of all the information on the web. As the web expanded, this became impossibly labour-intensive and the second chapter was heralded by the arrival of automation – search engines such as Alta Vista and Inktomi, which sent out software robots to crawl around the web to garner pages of information which could be retrieved using conventional textual indexing.

The advent of page ranking, for which Google deserves much of the credit, marked chapter three and the beginning of modern search. Rather than simply returning an undifferentiated mass of pages to the inquirer, page ranking software sorted the information hierarchically, placing the most significant first. Type in “IBM” and you will get the computer giant’s home page at the top of the list.

But, as Mr Horowitz points out, the hierarchy is decided by webmasters, the controllers of website content: “The webmasters get to vote by proxy on what’s important to all of us; they cast everyone’s vote, so we are slaves to the webmaster’s idea of what is important.”

He goes on: “This next phase of personal and social search means that we will empower individuals with the privilege of voting on what’s important for them and expose that so communities and social networks and other groups can leverage that information. The intention is to base social search on open standards in order to spread its influence beyond the Yahoo environment.”

It smacks a little of “expert systems” ideas, prevalent in the 1990s, where the knowledge and expertise of individuals would be captured and stored in a computer and retrieved on demand. Mr Horowitz agrees that many of the artificial intelligence specialists behind expert systems are now working on social search.

The fact remains that Yahoo faces a challenge in persuading the public that it has not slipped behind Google and Microsoft in the quality of its technology. Furthermore, it has no monopoly on ways to make the internet more than a simple repository of information. Mr Horowitz, however, has no doubt that social search is the future: “You don’t need social search to find the population of London – you do to find a plumber,” he claims. Now that could be a search too far, even for Yahoo.