Data Modeling

Data modeling is the work of deciding, in advance and on purpose, what things a database will hold, what facts about them matter, and how those facts connect.

11 frameworks, from 1966 to today · Databases · Computer Science

still a working view ended · each leads to its entry below

Overview

Data modeling is the work of deciding, in advance and on purpose, what things a database will hold, what facts about them matter, and how those facts connect1. The output is rarely a program; it is a description — boxes and arrows, tables and keys, a JSON schema, a diagram in a wiki — from which databases, programs, reports and interfaces are then built. What is at stake is everything downstream: a model that names the wrong things, or refuses to allow a fact to change, produces systems that are expensive to correct and sometimes wrong in ways nobody notices. The history of the field is a sequence of attempts to find a description precise enough for a machine and natural enough for the people who know the business, and each attempt was provoked by the failure of the one before.

Files, hierarchies, and the first database models

Before the 1960s, "data modeling" mostly meant laying out records on tape or disk: fixed fields, fixed lengths, and programs that knew exactly where each byte lived. The arrival of direct-access storage made something new possible — a store that programs could navigate rather than scan — and two competing answers to how to describe it dominated the decade.

The hierarchical data model, most famously realized in IBM's Information Management System (IMS), released in the late 1960s for the Apollo program's bill-of-materials data, organized records as a tree. Each parent could have many children; each child had exactly one parent2. For genuinely nested data — a company's divisions, departments and employees; a parts explosion — this matched the world. Its limitation was equally structural: a part used in two assemblies had to be stored twice or linked awkwardly, and a query "find all parts supplied by X" had to climb the tree from a fixed starting point.

The network data model, codified by the Conference on Data Systems Languages (CODASYL) Data Base Task Group report of 1969 and implemented in systems such as IDMS and IDS, generalized the tree to a graph. Records were connected by named sets, each with one owner record and many member records, and a record could be a member of several sets3. This solved the duplication problem but pushed the burden onto programmers: to ask a question you wrote a navigation program that walked pointers from record to record, and that program encoded the physical layout. Change the model and the programs broke. Both models shared a deeper assumption — that the description of data and the storage structure were essentially the same thing — and this is the assumption the next idea attacked.

The relational model: data described by values, not pointers

In 1970 Edgar F. Codd, a mathematician working at IBM's San Jose research laboratory, published "A Relational Model of Data for Large Shared Data Banks." Codd's claim was radical in its simplicity: describe all data as relations — tables of rows and columns — and let users ask questions in a declarative language that names the data they want, without any knowledge of how it is stored or reachable4. A relation is just a set of tuples; there are no pointers and no prescribed access paths4. Anything expressible in the network model, Codd argued, could be expressed relationally, and the description would be independent of the machine.

The model's practical breakthrough came with the query language. Codd proposed a relational calculus in the early 1970s; IBM's System R project, and in parallel the University of California, Berkeley's Ingres project, built working prototypes, and the language that emerged from System R — Structured English Query Language, later SQL — became the interface through which most of the world would ever touch a database. The commercial turning point was Oracle's first release in 1979, followed by IBM's SQL/DS and DB2 in the early 1980s. The relational model did not merely win; it became the default assumption of the industry for the next three decades5.

What made it persuasive was not elegance alone. A declarative query could be optimized by the system — the same question could be answered by different physical strategies, and the machine chose. That separation of logical description from physical execution is the founding idea of modern data modeling, and almost every later development in this history is either an extension of it or a reaction against its constraints.

Normalization: designing tables that behave

Once data lived in tables, a new question appeared: which tables? A badly designed table — customer address repeated on every order line — caused update anomalies, where changing one fact required changing many rows and risking inconsistency. Codd answered with normalization theory, beginning in 1971 and refined through the decade. The idea was to define, formally, the dependencies among attributes and then decompose tables so that each fact was stored once6.

A functional dependency is the statement that one set of attributes determines another: given a customer number, the customer's city is fixed. From such dependencies Codd and later theorists derived normal forms — first, second, third — each a condition on how dependencies are arranged7. Third normal form, roughly, says every non-key attribute depends on the whole key and nothing but the key7. A landmark clarification came from Ronald Fagin's work on multivalued dependencies and fourth normal form in the late 1970s, and further normal forms followed. The practical payoff was concrete: a schema in third normal form could not produce the anomaly where a customer's address is updated in one row and left stale in another7.

Normalization gave data modeling its first genuine method — a procedure with a theory behind it — and it remains the backbone of transactional database design. But it was also a theory about avoiding redundancy, not about capturing meaning. Two schemas could both be perfectly normalized and describe the business differently, and nothing in the normal forms said which was right.

Semantic models and the entity-relationship diagram

The 1970s produced a wave of semantic data models that tried to close that gap by building meaning into the description itself. The best known is Peter Chen's entity-relationship model, published in 19768. Chen proposed that a database be designed by first identifying entities — things that exist independently, such as a customer or a product — and relationships among them, with attributes attached to each, and then mapping that conceptual picture onto relational tables. The ER diagram, boxes for entities and diamonds for relationships, became and remains the most widely used picture in the field: it is how designers talk to domain experts who will never read a normal form.

Other semantic models explored related territory: the Semantic Data Model of Michael Hammer and Dennis McLeod (1981), which introduced the idea of a class with subclasses inheriting attributes; SDM's successor and the functional data model of David Shipman; and work on generalization and aggregation as modeling primitives. Their shared conviction was that a data model should express what the designer believes about the world, not just what fits the tables1.

Dimensional modeling: designing for questions, not for storage

Meanwhile a different pressure was building. By the late 1980s organizations had accumulated operational databases optimized for recording transactions, and analysts wanted to ask aggregate questions — sales by region by month by product — that these schemas answered slowly and awkwardly. Bill Inmon articulated the data warehouse in the early 1990s as a subject-oriented, integrated, time-variant store separate from operations, and Ralph Kimball supplied the design method that made it buildable: dimensional modeling.

Kimball's approach, laid out in The Data Warehouse Toolkit (1996), inverted the normalization instinct. Instead of decomposing to avoid redundancy, design around the business process being measured, with a central fact table holding numeric measures (quantity sold, revenue) and dimension tables describing the context — product, customer, time, store9. A star schema is one fact table surrounded by dimensions; a snowflake schema normalizes the dimensions further910. The method accepted redundancy in exchange for queries that were simple to write and fast to run. Kimball's dimensional bus architecture and Inmon's corporate information factory — an enterprise-wide normalized warehouse feeding dependent data marts — were the two poles of warehouse design debate through the late 1990s, and many organizations ran hybrids.

Objects, and the trouble with impedance

As programming moved to object-oriented languages in the 1980s and 1990s, modelers faced a mismatch: programs thought in objects with identity, encapsulation and inheritance; databases thought in tables of values. Object-oriented data modeling addressed this directly. Object database systems and the Object Data Management Group's standards gave objects persistent identity and allowed references between them, and modeling notations such as the Unified Modeling Language's class diagrams became the standard way to draw a domain model.

Object-oriented databases found niches in engineering and telecommunications but never displaced relational systems. The more consequential outcome was the object-relational hybridization: SQL:1999 added user-defined types, inheritance and complex values to the standard, and relational products absorbed enough object thinking to blunt the need for a separate class of database. The impedance mismatch was managed rather than solved, and it remains a live design tension wherever an application's object model and its storage schema diverge.

NoSQL: modeling for scale and shape

The late 2000s brought a shock to the relational consensus. Web-scale systems found that a single relational schema with strict normalization and joins was difficult to spread across hundreds of commodity machines, and that some data — a document, a social graph — did not decompose naturally into tables. NoSQL data modeling answered with models built around access patterns rather than around normalized facts.

The key-value store modeled data as a map from key to opaque value; the document model stored nested JSON-like structures with the whole document as the unit of retrieval; the wide-column model, popularized by Google's Bigtable paper (2006) and Cassandra, organized data into rows with sparse, dynamically named columns; the graph model stored entities as nodes and relationships as edges with their own attributes, which made traversal queries natural. The modeling discipline changed accordingly: instead of asking "what are the entities and dependencies," the NoSQL designer asks "what queries will the application run, and how do I lay out data so each is a single lookup?" Denormalization, duplication and precomputation became legitimate tools, with consistency handled by the application.

The commercial rise of these systems, and the eventual emergence of NewSQL systems that recombined SQL and horizontal scale, made the field plural. Relational, document, graph and columnar models now coexist in the same architecture, and "polyglot persistence" — choosing a model per workload — is an accepted practice.

Contracts, registries, and the governance turn

The most recent shift is less a new way to draw data than a new way to enforce agreement. As systems multiplied and data flowed between teams, the model stopped being an internal design artifact and became an interface between producers and consumers. Data contracts formalize this: the schema, semantics, quality expectations and ownership of a published dataset are agreed in writing, and breaking changes require a versioned negotiation. Schema registries, such as those built around Apache Avro and Confluent's Schema Registry, make it mechanical — producers register their schema, consumers fetch compatible versions, and the registry refuses changes that violate declared compatibility rules. The lineage is direct: this is Codd's insistence on describing data independently of programs, applied across organizational boundaries instead of within one system.

Where the field stands

Today the main approaches all descend from the turning points above. Entity-relationship modeling and normalization remain the default for transactional systems; dimensional modeling remains the default for analytics, though columnar storage and lakehouse architectures have blurred the warehouse boundary; document, graph and key-value models are standard options where access patterns justify them; object-relational mapping tools mediate between object models and tables; and contract-and-registry tooling is extending the discipline to streaming and inter-team data exchange. The open questions are the old ones in new clothing: how to keep a model faithful to a changing business, how to reconcile a conceptual model with several physical stores, and how to describe semantics — meaning, not just structure — well enough that machines can validate it. Semantic modeling was an unfinished project in the 1970s, and it is unfinished still.

References

  1. Data modeling (Overview, Semantic data modeling), Wikipedia
  2. Hierarchical database model, Wikipedia
  3. Network model (History), Wikipedia
  4. Relational model, Wikipedia
  5. Database (History), Wikipedia
  6. Database normalization, Wikipedia
  7. Third normal form, Wikipedia
  8. Entity–relationship model, Wikipedia
  9. Star schema, Wikipedia
  10. Snowflake schema, Wikipedia

The frameworks of Data Modeling

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1966-1985

    Hierarchical database model

    How should nested data be stored and navigated?

    Fly to its tower in Nousfarer

  2. 1969-1985

    Network model

    How can records be linked in a graph?

    Fly to its tower in Nousfarer

  3. 1970-Present

    Relational model

    How can data be described independently of storage?

    Fly to its tower in Nousfarer

  4. 1971-Present

    Database normalization

    How should tables be designed to avoid anomalies?

    Fly to its tower in Nousfarer

  5. 1976-Present

    Entity-relationship model

    How can entities and relationships be modeled conceptually?

    Fly to its tower in Nousfarer

  6. 1976-1995

    Semantic data model

    How can meaning be built into data models?

    Fly to its tower in Nousfarer

  7. 1985-2005

    Object-oriented data modeling

    How should objects be stored in databases?

    Fly to its tower in Nousfarer

  8. 1992-2010

    Corporate information factory

    How should an enterprise warehouse be structured?

    Fly to its tower in Nousfarer

  9. 1996-Present

    Dimensional modeling

    How should data be modeled for analytical queries?

    Fly to its tower in Nousfarer

  10. 2006-Present

    NoSQL data modeling

    How should data be modeled for distributed systems?

    Fly to its tower in Nousfarer

  11. 2018-Present

    Data contracts and schema registries

    How can data schemas be enforced across systems?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Hierarchical database modelNetwork modelRelational modelDatabase normalizationEntity-relationship modelSemantic data modelObject-oriented data modelingCorporate information factoryDimensional modelingNoSQL data modelingData contracts and schema registries

Beyond the field

Where Data Modeling's ideas came from in other fields, and what they led to there: the strongest 12 of 61 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Data Modeling is a planet in Nousfarer

Its city has a tower for each of these 11 frameworks. Land, walk in, and climb one concept at a time.

Land on Data Modeling

Related fields

Fields its ideas are linked to

More in Databases

See where Data Modeling sits on the Map of Human Knowledge