By Dr. Eric Glover, Searchme’s Classification Architect. Eric is responsible for the design and implementation of Searchme’s categories feature, a seemingly simple tool that springs from an exciting area of artificial intelligence (AI) research and development.

So, now that we actually have a mapping from all URLs to all categories for our dynamic ontology – considered by some to be the holy grail of web-scale machine learning – the question becomes: What do we do with this data? We’ve shown that it’s feasible (and we encourage you to test us), but what are we doing with it to make search better for our users?

So far, we’ve chosen one possible use for our ontology – presenting categories that help disambiguate queries and get users to the information they want more quickly. Currently, in our beta, we have more than 200 categories. Some are subjects like “golf”, while others are page types like “blogs” or “political news” (which were chosen to represent how users actually use the Web as opposed to how an academic might divide up a domain).

Each category is thoroughly tested to ensure sufficient (but nothing is perfect) accuracy. While testing, it’s exciting to me when random or obscure queries come back with perfect category suggestions. I have also seen cases where the ontology was not ideal or a category definition just didn’t work. For example, sites listing current spreads for a football game previously came back as “gambling and casinos”. Using our system, it took very little time to change the definition in the ontology and apply the changes (thus reclassifying our entire index). In addition, this work was done by a data analyst/ontologist, with no engineering resources required. This is only possible with a powerful system that supports our dynamic ontology and the ability to rapidly train and classify.

Every day I wake up excited to go to the office. I get to work with experts in the area of web search and machine learning to further enhance what we can do. We are already planning to grow the number of exposed categories, improve our accuracy, and further reduce the training costs of new verticals. We’ve got a long way to go, but I think this is the start of something that will radically change web search and the expectations of large-scale search engines. It has already been an exciting ride, and we have barely begun.

This concludes our series on ‘The Future of Web Search’, By Dr. Eric Glover, Searchme’s Classification Architect. Please check back as other Searchme team members will be posting on the topic as well.

This entry was posted on Friday, May 2nd, 2008 at 2:00 am and is filed under Artificial Intelligence, The Future of Search. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.