A concept search (or conceptual search) is an automated information retrieval method that is used to search electronically stored unstructured text (for example, digital archives, email, scientific literature, etc.) for information that is conceptually similar to the information provided in a search query. In other words, the ideas expressed in the information retrieved in response to a concept search query are relevant to the ideas contained in the text of the query.
__TOC__
Development Concept search techniques were developed because of limitations imposed by classical Boolean keyword search technologies when dealing with large, unstructured digital collections of text. Keyword searches often return results that include many non-relevant items (false positives) or that exclude too many relevant items (false negatives) because of the effects of synonymy and polysemy. Synonymy means that one of two or more words in the same language have the same meaning, and polysemy means that many individual words have more than one meaning.
Polysemy is a major obstacle for all computer systems that attempt to deal with human language. In English, the most frequently used terms have several common meanings. For example, the word fire can mean: a combustion activity; to terminate employment; to launch, or to excite (as in fire up). For the 200 most-polysemous terms in English, the typical verb has more than twelve common meanings, or senses. The typical noun from this set has more than eight common senses. For the 2000 most-polysemous terms in English, the typical verb has more than eight common senses and the typical noun has more than five.Bradford, R. B., Word Sense Disambiguation, Content Analyst Company, LLC, U.S. Patent 7415462, 2008.
In addition to the problems of polysemous and synonymy, keyword searches can exclude inadvertently misspelled words as well as the variations on the stems (or roots) of words (for example, strike vs. striking). Keyword searches are also susceptible to errors introduced by optical character recognition (OCR) scanning processes, which can introduce random errors into the text of documents (often referred to as noisy text) during the scanning process.
A concept search can overcome these challenges by employing word sense disambiguation (WSD),R. Navigli, Word Sense Disambiguation: A Survey, ACM Computing Surveys, 41(2), 2009. and other techniques, to help it derive the actual meanings of the words, and their underlying concepts, rather than by simply matching character strings like keyword search technologies.
Approaches In general, information retrieval research and technology can be divided into two broad categories: semantic and statistical. Information retrieval systems that fall into the semantic category will attempt to implement some degree of syntactic and semantic analysis of the natural language text that a human user would provide (also see computational linguistics). Systems that fall into the statistical category will find results based on statistical measures of how closely they match the query. However, systems in the semantic category also often rely on statistical methods to help them find and retrieve information.Greengrass, E., Information Retrieval: A Survey, 2000.
Efforts to provide information retrieval systems with semantic processing capabilities have basically used three approaches:
Auxiliary structures Local co-occurrence statistics Transform techniques (particularly matrix decompositions)
Auxiliary structures A variety of techniques based on artificial intelligence (AI) and natural language processing (NLP) have been applied to semantic processing, and most of them have relied on the use of auxiliary structures such as controlled vocabularies and ontologies. Controlled vocabularies (dictionaries and thesauri), and ontologies allow broader terms, narrower terms, and related terms to be incorporated into queries.Dubois, C., The Use of Thesauri in Online Retrieval, Journal of Information Science, 8(2), 1984 March, pp. 63-66. Controlled vocabularies are one way to overcome some of the most severe constraints of Boolean keyword queries. Over the years, additional auxiliary structures of general interest, such as the large synonym sets of WordNet, have been constructed.Miller, G., Special Issue, WordNet: An On-line Lexical Database, Intl. Journal of Lexicography, 3(4), 1990. It was shown that concept search that is based on auxiliary structures, such as WordNet, can be efficiently implemented by reusing retrieval models and data structures of classical information retrieval.Fausto Giunchiglia, Uladzimir Kharkevich, and Ilya Zaihrayeu. Concept Search , In Proceedings of European Semantic Web Conference, 2009. Later approaches have implemented grammar to expand the range of semantic constructs. The creation of data models that represent sets of concepts within a specific domain (domain ontologies), and which can incorporate the relationships among terms, has also been implemented in recent years.
Handcrafted controlled vocabularies contribute to the efficiency and comprehensiveness of information retrieval and related text analysis operations, but they work best when topics are narrowly defined and the terminology is standardized. Controlled vocabularies require extensive human input and oversight to keep up with the rapid evolution of language. They also are not well suited to the growing volumes of unstructured text covering an unlimited number of topics and containing thousands of unique terms because new terms and topics need to be constantly introduced. Controlled vocabularies are also prone to capturing a particular worldview at a specific point in time, which makes them difficult to modify if concepts in a certain topic area change.Bradford, R. B., Why LSI? Latent Semantic Indexing and Information Retrieval, White Paper, Content Analyst Company, LLC, 2008.
Local co-occurrence statistics Information retrieval systems incorporating this approach counts the number of times that groups of terms appear together (co-occur) within a sliding window of terms or sentences (for example, ± 5 sentences or ± 50 words) within a document. It is based on the idea that words that occur together in similar contexts have similar meanings. It is local in the sense that the sliding window of terms and sentences used to determine the co-occurrence of terms is relatively small.
This approach is simple, but it captures only a small portion of the semantic information contained in a collection of text. At the most basic level, numerous experiments have shown that approximately only a quarter of the information contained in text is local in nature.Landauer, T., and Dumais, S., A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge, Psychological Review, 1997, 104(2), pp. 211-240. In addition, to be most effective, this method requires prior knowledge about the content of the text, which can be difficult with large, unstructured document collections.
Transform techniques Some of the most powerful approaches to semantic processing are based on the use of mathematical transform techniques. Matrix decomposition techniques have been the most successful. Some widely used matrix decomposition techniques include the following:Skillicorn, D., Understanding Complex Datasets: Data Mining with Matrix Decompositions, CRC Publishing, 2007.
Independent component analysis Semi-discrete decomposition Non-negative matrix factorization Singular value decomposition
Matrix decomposition techniques are data-driven, which avoids many of the drawbacks associated with auxiliary structures. They are also global in nature, which means they are capable of much more robust information extraction and representation of semantic information than techniques based on local co-occurrence statistics.
Independent component analysis is a technique that creates sparse representations in an automated fashion,Honkela, T., Hyvarinen, A. and Vayrynen, J. WordICA - Emergence of linguistic representations for words by independent component analysis. Natural Language Engineering, 16(3):277-308, 2010 and the semi-discrete and non-negative matrix approaches sacrifice accuracy of representation in order to reduce computational complexity.
Singular value decomposition (SVD) was first applied to text at Bell Labs in the late 1980s. It was used as the foundation for a technique called latent semantic indexing (LSI) because of its ability to find the semantic meaning that is latent in a collection of text. At first, the SVD was slow to be adopted because of the resource requirements needed to work with large datasets. However, the use of LSI has significantly expanded in recent years as earlier challenges in scalability and performance have been overcome. and even open sourced.Gensim open source software LSI is being used in a variety of information retrieval and text processing applications, although its primary application has been for concept searching and automated document categorization.Dumais, S., Latent Semantic Analysis, ARIST Review of Information Science and Technology, vol. 38, Chapter 4, 2004.
Uses eDiscovery – Concept-based search technologies are increasingly being used for Electronic Document Discovery (EDD or eDiscovery) to help enterprises prepare for litigation. In eDiscovery, the ability to cluster, categorize, and search large collections of unstructured text on a conceptual basis is much more efficient than traditional linear review techniques. Concept-based searching is becoming accepted as a reliable and efficient search method that is more likely to produce relevant results than keyword or Boolean searches.Magistrate Judge John M. Facciola of the U.S. District Court for the District of Washington, D.C. Disability Rights Council v. Washington Metropolitan Transit Authority, 242 FRD 139 (D. D.C. 2007), citing George L. Paul & Jason R. Baron, "Information Inflation: Can the Legal System Adapt?" 13 Rich. J.L. & Tech. 10 (2007).
Enterprise Search and Enterprise Content Management (ECM) – Concept search technologies are being widely used in enterprise search. As the volume of information within the enterprise grows, the ability to cluster, categorize, and search large collections of unstructured text on a conceptual basis has become essential. In 2004 the Gartner Group estimated that professionals spend 30 percent of their time searching, retrieving, and managing information.Laplanche, R., Delgado, J., Turck, M., Concept Search Technology Goes Beyond Keywords, Information Outlook, July 2004. The research company IDC found that a 2,000-employee corporation can save up to $30 million per year by reducing the time employees spend trying to find information and duplicating existing documents.