Knowledge extraction is the creation of knowledge from structured (relational databases, XML) and unstructured (text, documents, s) sources. The resulting knowledge needs to be in a machine-readable and machine-interpretable format and must represent knowledge in a manner that facilitates inferencing. Although it is methodically similar to information extraction (IE) in natural language processing (NLP) and extract, transform, load (ETL), the main criterion is that the extraction result goes beyond the creation of structured information or the transformation into a relational schema. It requires either the reuse of existing formal knowledge (reusing identifiers or ontologies) or the generation of a schema based on the source data.
The RDB2RDF W3C group was standardizing a language for extraction of resource description frameworks (RDF) from relational databases. Another popular example for knowledge extraction is the transformation of Wikipedia into structured data and also the mapping to existing knowledge (see DBpedia and Freebase).
Overview After the standardization of knowledge representation languages such as RDF and OWL, much research has been conducted in the area, especially regarding transforming relational databases into RDF, identity resolution, knowledge discovery and ontology learning. The general process uses traditional methods from information extraction and extract, transform, and load (ETL), which transform the data from the sources into structured formats. So understanding how the interact and learn from each other.
The following criteria can be used to categorize approaches in this topic (some of them only account for extraction from relational databases): Source Which data sources are covered: Text, Relational Databases, XML, CSV Exposition How is the extracted knowledge made explicit (ontology file, semantic database)? How can you query it? Synchronization Is the knowledge extraction process executed once to produce a dump or is the result synchronized with the source? Static or dynamic. Are changes to the result written back (bi-directional) Reuse of vocabularies The tool is able to reuse existing vocabularies in the extraction. For example, the table column 'firstName' can be mapped to foaf:firstName. Some automatic approaches are not capable of mapping vocab. Automatization The degree to which the extraction is assisted/automated. Manual, GUI, semi-automatic, automatic. Requires a domain ontology A pre-existing ontology is needed to map to it. So either a mapping is created or a schema is learned from the source (ontology learning).
Examples
Entity linking DBpedia Spotlight, OpenCalais, Dandelion dataTXT, the Zemanta API, Extractiv and PoolParty Extractor analyze free text via named-entity recognition and then disambiguates candidates via name resolution and links the found entities to the DBpedia knowledge repository (Dandelion dataTXT demo or DBpedia Spotlight web demo or PoolParty Extractor Demo).
President Obama called Wednesday on Congress to extend a tax break for students included in last year's economic stimulus package, arguing that the policy provides more generous assistance.
As President Obama is linked to a DBpedia LinkedData resource, further information can be retrieved automatically and a Semantic Reasoner can for example infer that the mentioned entity is of the type Person (using FOAF (software)) and of type Presidents of the United States (using YAGO). Counter examples: Methods that only recognize entities or link to Wikipedia articles and other targets that do not provide further retrieval of structured data and formal knowledge.
Relational databases to RDF Triplify, D2R Server, Ultrawrap , and Virtuoso RDF Views are tools that transform relational databases to RDF. During this process they allow reusing existing vocabularies and ontologies during the conversion process. When transforming a typical relational table named users, one column (e.g.name) or an aggregation of columns (e.g.first_name and last_name) has to provide the URI of the created entity. Normally the primary key is used. Every other column can be extracted as a relation with this entity. Then properties with formally defined semantics are used (and reused) to interpret the information. For example, a column in a user table called marriedTo can be defined as symmetrical relation and a column homepage can be converted to a property from the FOAF Vocabulary called foaf:homepage, thus qualifying it as an inverse functional property. Then each entry of the user table can be made an instance of the class foaf:Person (Ontology Population). Additionally domain knowledge (in form of an ontology) could be created from the status_id, either by manually created rules (if status_id is 2, the entry belongs to class Teacher ) or by (semi)-automated methods (ontology learning). Here is an example transformation:
Name marriedTo homepage status_id Peter Mary https://example.org/Peters_page 1 Claus Eva https://example.org/Claus_page 2 :Peter :marriedTo :Mary . :marriedTo a owl:SymmetricProperty . :Peter foaf:homepage <https://example.org/Peters_page> . :Peter a foaf:Person . :Peter a :Student . :Claus a :Teacher .
Extraction from structured sources to RDF
1:1 Mapping from RDB Tables/Views to RDF Entities/Attributes/Values
When building a RDB representation of a problem domain, the starting point is frequently an entity-relationship diagram (ERD). Typically, each entity is represented as a database table, each attribute of the entity becomes a column in that table, and relationships between entities are indicated by foreign keys. Each table typically defines a particular class of entity, each column one of its attributes. Each row in the table describes an entity instance, uniquely identified by a primary key. The table rows collectively describe an entity set. In an equivalent RDF representation of the same entity set: Each column in the table is an attribute (i.e., predicate) Each column value is an attribute value (i.e., object) Each row key represents an entity ID (i.e., subject) Each row represents an entity instance Each row (entity instance) is represented in RDF by a collection of triples with a common subject (entity ID).
So, to render an equivalent view based on RDF semantics, the basic mapping algorithm would be as follows: create an RDFS class for each table convert all primary keys and foreign keys into IRIs assign a predicate IRI to each column assign an rdf:type predicate for each row, linking it to an RDFS class IRI corresponding to the table for each column that is neither part of a primary or foreign key, construct a triple containing the primary key IRI as the subject, the column IRI as the predicate and the column's value as the object.
Early mentioning of this basic or direct mapping can be found in Tim Berners-Lee's comparison of the ER model to the RDF model.
Complex mappings of relational databases to RDF
The 1:1 mapping mentioned above exposes the legacy data as RDF in a straightforward way, additional refinements can be employed to improve the usefulness of RDF output respective the given Use Cases. Normally, information is lost during the transformation of an entity-relationship diagram (ERD) to relational tables (Details can be found in object-relational impedance mismatch) and has to be reverse engineered. From a conceptual view, approaches for extraction can come from two directions. The first direction tries to extract or learn an OWL schema from the given database schema. Early approaches used a fixed amount of manually created mapping rules to refine the 1:1 mapping. More elaborate methods are employing heuristics or learning algorithms to induce schematic information (methods overlap with ontology learning). While some approaches try to extract the information from the structure inherent in the SQL schema (analysing e.g. foreign keys), others analyse the content and the values in the tables to create conceptual hierarchies (e.g. a columns with few values are candidates for becoming categories). The second direction tries to map the schema and its contents to a pre-existing domain ontology (see also: ontology alignment). Often, however, a suitable domain ontology does not exist and has to be created first.
XML
As XML is structured as a tree, any data can be easily represented in RDF, which is structured as a graph. XML2RDF is one example of an approach that uses RDF blank nodes and transforms XML elements and attributes to RDF properties. The topic however is more complex as in the case of relational databases. In a relational table the primary key is an ideal candidate for becoming the subject of the extracted triples. An XML element, however, can be transformed - depending on the context- as a subject, a predicate or object of a triple. XSLT can be used a standard transformation language to manually convert XML to RDF.
Survey of methods / tools
Name Data Source Data Exposition Data Synchronisation Mapping Language Vocabulary Reuse Mapping Automat. Req. Domain Ontology Uses GUI A Direct Mapping of Relational Data to RDF Relational Data SPARQL/ETL dynamic false automatic false false CSV2RDF4LOD CSV ETL static RDF true manual false falseCoNLL-RDFTSV, CoNLLSPARQL/ RDF streamstaticnonetrueautomatic (domain-specific, for use cases in language technology, preserves relations between rows)falsefalse Convert2RDF Delimited text file ETL static RDF/DAML true manual false true D2R Server RDB SPARQL bi-directional D2R Map true manual false false DartGrid RDB own query language dynamic Visual Tool true manual false true DataMaster RDB ETL static proprietary true manual true true Google Refine's RDF Extension CSV, XML ETL static semi-automatic false true Krextor XML ETL static xslt true manual true false MAPONTO RDB ETL static proprietary true manual true false METAmorphoses RDB ETL static proprietary xml based mapping language true manual false true MappingMaster CSV ETL static MappingMaster true GUI false true ODEMapster RDB ETL static proprietary true manual true true OntoWiki CSV Importer Plug-in - DataCube & Tabular CSV ETL static The RDF Data Cube Vocaublary true semi-automatic false true Poolparty Extraktor (PPX) XML, Text LinkedData dynamic RDF (SKOS) true semi-automatic true false RDBToOnto RDB ETL static false automatic, the user furthermore has the chance to fine-tune results false true RDF 123 CSV ETL static false false manual false true RDOTE RDB ETL static SQL true manual true true Relational.OWL RDB ETL static false automatic false false T2LD CSV ETL static false false automatic false false The RDF Data Cube Vocabulary Multidimensional statistical data in spreadsheets Data Cube Vocabulary true manual false TopBraid Composer CSV ETL static SKOS false semi-automatic false true Triplify RDB LinkedData dynamic SQL true manual false false Ultrawrap RDB SPARQL/ETL dynamic R2RML true semi-automatic false true Virtuoso RDF Views RDB SPARQL dynamic Meta Schema Language true semi-automatic false true Virtuoso Sponger structured and semi-structured data sources SPARQL dynamic Virtuoso PL & XSLT true semi-automatic false false VisAVis RDB RDQL dynamic SQL true manual true true XLWrap: Spreadsheet to RDF CSV ETL static TriG Syntax true manual false false XML to RDF XML ETL static false false automatic false false
Extraction from natural language sources