Semi-structured data is a form of structured data that is not rigidly structured. The data model associated with semi-structured data is interpreted when the data is read from a data storage medium or accessed in memory. The data model need not conform to a predefined database schema and is not optimized for succinct, low-overhead streaming or persistent storage as is normal with a relational database, graph database, data table, or graph. Instead, semi-structured data includes delimiter characters or tags to separate semantic elements or to describe relationships among records and fields within the data. The data is said to have a self-describing structure.
Semi-structured data may be human-generated or machine-generated. It may be part of an unstructured document such as narrative text or may encompass unstructured data such as free-form comments. It may be intended to be read by humans, to be shared between software services or applications whose data models are not fully consistent, or to be transmitted from sensors or other devices to computers. Optical character recognition in combination with intelligent document processing can convert semi-structured data appearing on printed pages into machine-ready formats such as JSON. Electronic data interchange uses standard segments, elements, and delimiters to transfer business transaction data between computers owned by different organizations. Metrics from an array of various sensors can be combined, as heterogeneous data without predefined schemas, in JSON format.
Human-generated sources
Human-generated semi-structured data includes text and information organized with tags, markers, or flexible layouts (such as forms or tables) created by people rather than by automated systems. Examples include medical charts, real property records, pay records, and structural metadata within publications.
Medical records
Clinical notes containing standardized dropdown selections can be mixed with free-text physician narratives. The Fast Healthcare Interoperability Resources (FHIR) standard is used for exchanging health care information electronically.
Patient intake forms featuring labeled fields (e.g., age, symptoms) and handwritten or typed comments can be digitally stored using JSON or FHIR standards.
Electronic health record logs can be formatted via standards like HL7 that group data into segments.
Real estate
Real estate records such as deed, mortgage, and lien documents can be processed as JSON objects.
Scanned or digital property lease documents often include irregular tables, hierarchical headers, and customized clauses that can be treated as semi-structured data.
Property listing descriptions represented via listing services include such information as square footage and room count along with free-form descriptive text.
Human-annotated zoning and appraisal filings also combine numeric values with descriptive text.
Pay records
Employee pay stubs feature fixed field labels (gross pay, deductions) along with descriptive text.
Human-entered spreadsheet logs track contractor hours and commission adjustments.
Corporate compensation disclosures are managed through tracking schemas like the Open Payments Search Tool associated with the Physician Payments Sunshine Act of 2010.
Publications
A book or periodical has structural metadata such as its table of contents, chapters, parts, and an index. These can be represented in JSON.
Bibliographical references conform to a structure, typically defined by the MLA Handbook, the APA, or the Chicago Manual of Style, that can be parsed based on sets of delimiters or using wildcards.
Machine-generated sources
Machine-generated semi-structured data sources are automated systems and devices that output information organized with tags, keys, or markers rather than rigid tabular schemas. Examples include server log files, IoT sensor metrics, event streams, NoSQL database engines, email systems, mapping, and geographic information systems.
System and server logs
Application outputs, error logs, and security audit trails can use text patterns or key-value pairs without a strict schema.
IoT and telemetry streams
Many smart devices, industrial sensors, and interconnected hardware components generate periodic metrics packaged in JSON or XML.
Web and API activity
Automated webhooks, clickstream tracking data, and web API response payloads are typically packaged in JSON.
Network & monitoring data
Packet inspection summaries, router statistics, and application performance management (APM) metrics provide semi-structured data in various formats.
NoSQL database engines
A NoSQL database management system, often used in big data and real-time applications, is designed to handle semi-structured data models, including key-value, document, column family, and graph formats.
Email headers
An email consists of headers and a body. An email header conforms to a standard key-value format that includes a field name followed by a colon that serves as a delimiter.
Mapping and geographic information systems
GeoJSON is a standard for encoding geographic data structures, such as points, lines, and polygons, into a JSON-based format used to represent geographic features, such as locations, roads, boundaries, and their names. OpenStreetMap stores semi-structured data using a flexible tag-based data model built on elements such as nodes, ways, and relations.
Formats for machine-generated semi-structured data
Application-specific or domain-specific formats
Domain-specific datasets are collections of data specific to a particular field. These datasets focus on specific types of information relevant to a domain such as health care or finance. The data model or format used for domain-specific data may be tailored for use cases within that domain.
An application program or a database management system may work with semi-structured data without regard to its interoperability with other software, but a semi-structured data model that exists for its own sake may be unnecessarily cumbersome. Such systems have been designed so that data associated with object-oriented software is serialized into a semi-structured format, allowing for the concept that objects of a class may have varying attributes. The data associated with the objects of a class may be grouped, and the attributes' order is not important. In some object-oriented database management systems, such as ObjectStore, the format for semi-structured data storage is specific to the system.
Database management systems that lack interoperability have fallen out of vogue. Standardized formats for semi-structured data have arisen to serve as a basis for interoperability, for scalability, and for adaptability to schema evolution.
XML and OEM In the 1990s, XML, and the Object Exchange Model (OEM)Stanford Universities Lore DBMS were created as generalized formats for semi-structured data. XML has been popularized by web services developed utilizing SOAP principles of decentralization, transport independence, and XML-based structured messaging. XML suffered from the impression that it lacked structural rigor at the same functional level as relational schemas. Once JSON became popular, its compatibility with JavaScript and its relatively human-readable format handicapped the widespread use of XML. Nevertheless, an XML document can be as rigorously structured as a database schema and processed by both commercial and custom software programs. XML data can have a reasonably human-readable flow and hierarchy as well as rigorous relational structure and data typing.
The concept of XML as human-readable, however, can only be taken so far. Some implementations, such as the XML representation of the contents of a Microsoft Word document, as implemented in Office 2007 and later versions, utilize dozens or even hundreds of different kinds of tags that reflect a particular problem domain - in Word's case, formatting at the character and paragraph and document level, definitions of styles, inclusion of citations, etc. - which are nested within each other in complex ways. Understanding even a portion of such an XML document by reading it, let alone catching errors in its structure, is impossible without a deep prior understanding of the specific XML implementation, along with assistance by software that understands the XML schema that has been employed. Such text is not "human-understandable" any more than a book written in Swahili, which uses the Latin alphabet, would be to a reader who does not know a word of that language but recognizes only its alphabet. Because XML tags express shared vocabularies, they are symbols that are meaningless to a person unfamiliar with the domain in which they apply.
YAML
YAML (YAML Ain't Markup Language) is a human-readable data serialization format designed for data exchange between differing programming languages. It is used for configuration files and for representing complex nested data structures.
TOML
TOML is designed as a configuration file format for mapping file elements to a hash table. Implementations are available for over thirty programming lanauges.
INI
The INI format arranges key-value pairs in named sections. It is used in configuration files for applications that run on Windows and other operating systems.
Log file formats
Log files are used for a range of purposes such as performance monitoring and debugging. They may include semi-structured data. Various log formats are in common use.
Apache Avro, Parquet, and ORC
The Apache Software Foundation provides open source file formats for data storage and retrieval. Each format is designed for a specific data orientation. Apache Avro is a row-based format used for streaming data and append-only logs. Apache Parquet is a column-based format used for data lake repositories. Apache ORC is an alternate columnar data format used for data warehouses.
Delimiter-separated values
Files containing text delimited by newlines, commas (as in a format known as CSV), semicolons, tabs, or other separators are used in data exchange, analytics, and integration.
JSON
JSON (JavaScript Object Notation) is an open standard format that uses human-readable text to represent and stream data objects. JSON has been popularized by web services developed using REST principles. Databases such as MongoDB and Apache CouchDB work natively with data stored in JSON format.
Pros and cons of semi-structured data formats
A database or standalone application may store or stream data in a succinct machine-readable format, or it may use a semi-structured format. Compared to a succinct format, a semi-structured format has certain advantages and disadvantages.
Advantages Developers persisting objects from their application to a database do not need to worry about object-relational impedance mismatch, but can often serialize objects via a light-weight library. The data representation has flexible schemas, self-describing tags or delimiters, and nested hierarchies, rather than any rigid format, which may simplify data models. Support for lists of objects, as needed for object-relational mapping, simplifies data models by avoiding messy translations of lists into a relational or graph data model. Choosing a commonly-used human-readable format over a succinct format may aid software debugging and maintenance, interoperability, and schema evolution. A semi-structured data format promotes rapid application development via the schema-on-read paradigm by letting applications store data without first modifying a database schema, offering flexibility, fast iteration, and ease of changing the data model.
Disadvantages