The Future of Web Search
Part Five: Towards A Large, Dynamic Ontology
By Dr. Eric Glover, Searchme’s Classification Architect. Eric is responsible for the design and implementation of Searchme’s categories feature, a seemingly simple tool that springs from an exciting area of artificial intelligence (AI) research and development.
Because it’s very difficult to create a deep, useful ontology that actually works and stands the test of time, it is understandable that search engines have moved towards the simple approach of “do you want news, images or blogs?” served with a side of “search only in English.” Is there a way to create a better ontology that actually works and is flexible over time?
The answer is yes. We have the ability to create a responsive ontology, one that is rapidly adaptable. With a dynamic ontology, if someone new becomes famous next week, we could create and apply a category for that person, instantly. Likewise, if an existing category changes or becomes obsolete, it is easy to adjust.
Being both deep (many categories) and flexible, however, makes it difficult to effectively map web pages onto such an ontology, because things are changing all the time. In addition, the rapidly changing definitions mean the potential maintenance costs may be prohibitive – especially if each change requires thousands of labeled examples and days of training. So how do you have a classification system on such a grand scale – one that makes business sense?
There are hundreds of academic papers on how to do text classification, but few methods are viable when applied to billions of web pages and hundreds or thousands of categories. Typically, they are too slow, not accurate enough, or too expensive to train/maintain.
How do we create a dynamic ontology that really works for large-scale web classification?
Next – Part Six: Creating A Large-Scale, Dynamic Ontology