Funny, that’s what I wrote on June 12, 2003, for Danny and Chris at Search Engine Watch.


Search has changed dramatically since the early AltaVista days under Digital Equipment. The pure search and technology-centric engines left standing have transformed themselves into direct-marketing businesses. Too bad AltaVista was so busy thinking of itself as a portal while Overture was inventing performance-based keyword search marketing, and Google was laser-focused on relevancy, laying down the groundwork for its successfully relevant AdWord paid inclusion program.

So what’s it going to take to beat Google? Here’s a look at some of the critical factors search engines need to address to be successful in today’s environment.

Relevancy, performance, index, and ease of use

Relevancy, index size and freshness, performance, and ease of use are still critical factors, but they are no longer sufficient to predict future success.

Google’s claim to fame, with a share of about 55% of usage according to OneStat.com, came from delivering the most relevant results of all. However, the relevancy gap between Google and other search engines like Inktomi or AlltheWeb now seems to have faded. Most major engines are now very good at serving very relevant results given the few clues they can glean from the query, typically made up just of one or two search keywords. Are search engines’ results as relevant as they will ever get?

Performance, or how quickly browsers display the results, has become a non-issue except for sporadic downtimes. The size of the web is growing continuously, so are indices of the major crawler-based engines. Spiders are racing through the web, refreshing their content more often than ever.

Portals have poured money into usability studies to find out what users want their results to look like. There is not a one-fit all look and feel, but search engines seem to have finally figured out that non-relevant clutter is bad and simplicity is good.

To remain on the edge, search engines have to continuously push the frontiers of innovation.

Users’ intent and personalized relevancy

The subjective nature of users’ intent when formulating queries is complex. Understanding the context is challenging, but two important factors, location and time, are under-exploited by the engines today. A user typing the query [pizza] from Chicago or Milan, at 11:00am or 9:00pm local time, doesn’t expect the same results. Besides, the local advertising market is considerable and better geo-based services are critical to some wireless devices.

Training the engines

Search engines should learn from our behaviors. Users often search on the same topic, typing recurring queries. Users don’t always click on the first result and often navigate back and forth selecting links from a results page. Search engines should be more proactive, learn for the benefit of individual users and become smarter over time.

Think of it as training the engines. The more you use them, the better advice you receive. There are issues to address, including privacy concerns and how to handle multiple users of a single PC, to name only a few. But these challenges these issues pose can be overcome.

Query disambiguation

Emerging concept-based clustering technologies, used in search engines such as Vivisimo, are doing wonders at allowing users to refine ambiguous queries. For example, searchers can discriminate between computer virus and medical virus from a more generic [virus] query, but they still have to do the work after the results are served.

Disambiguation technologies have not been fully leveraged yet. Andrei Broder, Distinguished Engineer and CTO of the Institute for Search and Text Analysis, part of the IBM Research Division, explains that queries can be divided into three categories: informational, navigational, and transactional.

Some queries are clearly transactional, such as [iPaq H1910]; others are clearly informational, such as [muscular atrophy], as others clearly are navigational, such as [NBA official site]. Some smart linguists could dissect and categorize these search terms and make better sense of the users’ intent.

Making results productive

Search engines should better follow through and make results more productive for users, notifying them of new relevant results for specific queries. Users often go to the same sites looking for new content. Search engines could monitor these changes for us. What a great opportunity for direct marketing businesses to establish that one-to-one marketing relationship, directly addressing users’ needs, and serve a relevant Google AdWord or Overture paid link in the notification email.

Vertical engines

The continued emergence of vertical search engines will increasingly fragment the market, eroding relative usage share away from Google. Crawling deeper through the invisible web, the indices of these topical engines have more depth for the subjects they cover. Findlaw.com, for example, allows users to retrieve proprietary content such as cases, opinions, and other legal reference material unavailable elsewhere.

Serving niche markets and a very targeted user bases, these sites are in a position to offer marketers better click conversion rates and, command higher cost-per-clicks than the general purpose engines.

Meta search engines

Meta search engines, such as Infospace’s Dogpile.com, Vivisimo.com, and Mamma.com have a very good shot at beating Google. Why wouldn’t the most relevant results from several of the best engines not be more relevant than the results of a single – even the best – crawler-based engine?

Meta search engines are not as technology-centric as search engines. But they have spent too much time trying to replicate single-source crawler-based engines and have been carrying a bad reputation for serving too many irrelevant paid links and obtrusive popup advertisements. Meta search engines should build on the differentiated and added value of aggregating results from the best sources.

Conclusion

The recent consolidation trends mean fewer players. Inevitably, these are also the fittest, with more negotiation power. Yahoo acquiring Overture would tip the balance of power, creating a formidable competitor to Google. MSN Search has certainly not played its hand yet. Redmond cannot feel too good about fueling Yahoo’s traffic and paid placement revenues.

More players and competitors will surface as more creative and sustainable business models emerge. Search players will increasingly focus on respective and distinct core competencies. Indexing the web is a complex task, as is researching smarter relevancy algorithms. Richer concept-based marketing tools will require more sophisticated skills.

A new model could very well emerge, where crawlers crawl and marketing firms target campaigns. Meta search engines could very well differentiate themselves providing real aggregation value, executing on relevancy and user experience, and emerge as the top search destinations.