Loading field map
A database is a structured collection of data, and a database management system (DBMS) is the software that defines, stores, retrieves, updates, and administers that data. The subfield of databases is concerned with the theory, design, implementation, and use of such systems. Its central challenge is to manage data reliably, efficiently, and securely over long periods, often for many concurrent users, while supporting complex queries and updates. The stakes are high because databases underpin nearly every modern information system, from banking and e-commerce to scientific research and social media. The field is not a single monolithic discipline but a landscape of distinct approaches, each addressing a different set of problems, with a shared foundation in a few core concepts.
The modern history of databases begins with the relational model, proposed by Edgar F. Codd in 1970. Before this, data was typically organized in hierarchical or network models, where the programmer navigated explicit links between records. Codd's insight was to organize data into tables (relations) of rows (tuples) and columns (attributes), with no predefined access paths. Data is retrieved by declaring what is wanted, not how to find it. This separation of logical structure from physical storage is the model's defining feature.
The relational model is built on a rigorous mathematical foundation: set theory and first-order logic. A relation is a set of tuples, and operations on relations—such as selection, projection, and join—are defined in terms of set operations. This formalism gives the model its power: queries are declarative, and the system's optimizer can choose an efficient execution plan without the user specifying one. The standard language for interacting with relational databases is SQL (Structured Query Language), which, despite its name, is used for both data definition and data manipulation. SQL's SELECT statement, with its FROM, WHERE, GROUP BY, and ORDER BY clauses, is a direct expression of relational algebra and calculus.
The relational model's success came from its combination of simplicity and power. It made data access easier for users and applications, and it provided a clear theoretical basis for the field. However, the model's purity was soon tempered by practical needs. The original model assumed that all data is atomic and that relations are sets (with no duplicate rows). Real systems introduced features like NULL values for missing data, duplicate rows, and automatic ordering, which technically violate the strict relational model but are essential for practical usability. This tension between theoretical elegance and engineering pragmatism is a recurring theme in the field.
A database is not just a static store; it must support concurrent updates from many users while maintaining consistency. This is the domain of transaction management. A transaction is a sequence of operations that is treated as a single logical unit. The field's central guarantee is captured by the ACID properties: Atomicity (all operations in a transaction succeed or none do), Consistency (a transaction transforms the database from one valid state to another), Isolation (concurrent transactions do not interfere with each other), and Durability (once a transaction commits, its effects survive system failures).
Achieving these properties is a major engineering and theoretical challenge. The two main techniques are locking and logging. Locking, typically using a protocol called two-phase locking, controls access to data items to ensure isolation. Logging, usually a write-ahead log, records every change so that the system can undo incomplete transactions (for atomicity) and redo committed ones after a crash (for durability). The theory of serializability—the idea that the result of concurrent transactions should be the same as if they ran in some serial order—provides the formal criterion for correctness.
The ACID model is the gold standard for correctness, but it comes at a cost. Strict isolation requires locks, which can reduce concurrency and throughput. This has led to a spectrum of alternatives. Weaker isolation levels, such as read committed or snapshot isolation, allow more concurrency by relaxing the guarantee of serializability, accepting that some anomalies may occur. The choice of isolation level is a trade-off between correctness and performance, and it is a decision that database administrators and application developers must make explicitly.
For decades, the relational model dominated, and the field's research agenda was largely about making relational systems faster, more scalable, and more reliable. This consensus began to fracture in the mid-2000s, driven by the needs of large-scale web applications and the availability of cheap, commodity hardware. The result was a family of systems collectively known as NoSQL (often interpreted as "Not Only SQL"). These systems are not a single school but a set of distinct approaches, each relaxing different aspects of the relational model to gain other advantages.
One major NoSQL family is the key-value store, which treats the database as a large hash table. Data is accessed by a unique key, and the value is an opaque blob. This model is extremely simple and highly scalable, but it offers no query language beyond key lookups and no support for relationships between data items. A second family is the document store, which organizes data into documents (typically JSON or XML). Documents are self-describing and can have nested structures, which maps naturally to how many applications model their data. Document stores often support secondary indexes and a limited query language, but they do not enforce a fixed schema. A third family is the wide-column store, which organizes data into tables with rows and columns, but where columns can vary by row and are grouped into column families. This model is designed for massive scale and analytical workloads, and it is closely associated with the Google Bigtable paper and its open-source implementation, Apache HBase. A fourth family is the graph database, which explicitly models data as nodes and edges, making it natural for social networks, recommendation engines, and other relationship-heavy domains. Graph databases support queries that traverse relationships, such as "find all friends of friends," which are awkward and slow in SQL.
The NoSQL movement was not a single paradigm but a rejection of the relational model's universality. Its proponents argued that the ACID guarantees and the relational schema were too restrictive for certain workloads, and that horizontal scalability (adding more machines) was more important than strict consistency. This led to the CAP theorem, a foundational result that states a distributed data system can guarantee at most two of three properties: Consistency (all nodes see the same data at the same time), Availability (every request receives a response), and Partition tolerance (the system continues to operate despite network failures). Since network partitions are unavoidable in distributed systems, the theorem forces a choice between consistency and availability during a partition. This theorem, while often simplified, frames the fundamental trade-off in distributed databases.
The NoSQL systems solved the scalability problem but often sacrificed the conveniences of SQL: declarative queries, joins, transactions, and a well-defined schema. This created a gap that a newer generation of systems, sometimes called NewSQL, attempted to fill. The goal of NewSQL is to provide the full power of SQL and ACID transactions while achieving the horizontal scalability of NoSQL systems.
NewSQL systems take various technical approaches. Some use a shared-nothing architecture, where data is partitioned across many nodes, and a distributed transaction coordinator ensures ACID guarantees across partitions. Others use a shared-disk or shared-memory architecture with sophisticated concurrency control. Some systems, like Google Spanner, use a combination of techniques, including synchronized clocks and a special protocol for distributed transactions, to achieve global consistency at scale. The key insight of NewSQL is that the relational model and ACID are not inherently incompatible with scalability; the challenge is purely an engineering one of building a distributed system that can execute SQL queries and transactions correctly and efficiently.
NewSQL is not a single school but a set of engineering responses to the perceived failures of both traditional relational systems and NoSQL systems. Its success is still being evaluated, and it has not fully replaced either approach. However, it has pushed the field toward a more nuanced view: the choice of database technology is not a binary between SQL and NoSQL, but a spectrum of trade-offs involving consistency, scalability, query expressiveness, and operational complexity.
The systems discussed so far are primarily designed for online transaction processing (OLTP), where the workload consists of many small, concurrent read and write operations. A different set of problems arises in online analytical processing (OLAP), where the workload consists of a few large, complex queries that scan and aggregate vast amounts of data. This distinction has led to a separate line of development.
The traditional solution for OLAP is the data warehouse, a separate database optimized for analytical queries. Data is extracted from operational systems, transformed (cleaned, integrated, and aggregated), and loaded into the warehouse (a process known as ETL). The warehouse is typically organized using a star or snowflake schema, where a central fact table contains measures (e.g., sales amount) and foreign keys to dimension tables (e.g., product, time, customer). This schema is designed to make aggregation queries efficient.
A key technical innovation for OLAP is the columnar store. Instead of storing data row by row, as in a traditional row-oriented DBMS, a columnar store stores each column separately. This has two major advantages for analytical workloads. First, a query that only needs a few columns can read only those columns from disk, avoiding the I/O cost of reading entire rows. Second, columnar data compresses much better because values in a column are often similar. Modern analytical databases, such as those used for business intelligence and data science, are almost universally columnar. This is a clear example of how a specific workload (OLAP) has driven a distinct technical approach, which is now a standard part of the database landscape.
The current database landscape is diverse and pluralistic. The relational model remains the default for most business applications, and SQL is the lingua franca of data access. NoSQL systems are well established for specific use cases, such as caching, session storage, and large-scale content management. NewSQL systems are gaining traction in domains that need both SQL and scale. Analytical databases, both in the form of on-premises data warehouses and cloud-based data platforms, are essential for decision support and business intelligence. The rise of cloud computing has also introduced the database-as-a-service model, where the operational burden of running a database is outsourced to a cloud provider.
Amid this diversity, several enduring questions continue to shape the field. One is the trade-off between consistency and availability in distributed systems, which the CAP theorem frames but does not resolve. Another is the tension between schema flexibility and data integrity: NoSQL systems offer flexibility but push the burden of validation to the application, while relational systems enforce integrity but require migration when the schema changes. A third question is how to handle the increasing volume, velocity, and variety of data, which has led to the development of specialized systems for streaming data, time-series data, and geospatial data. Finally, the field is grappling with the challenge of integrating databases with machine learning, both in terms of using databases to support ML pipelines and using ML techniques to optimize database internals.
The subfield of databases is thus best understood not as a single paradigm but as a set of interconnected approaches, each with its own assumptions, strengths, and limitations. The relational model provides the theoretical foundation and the dominant query language. Transaction management provides the guarantees of correctness. NoSQL systems explore the trade-offs of relaxing those guarantees for scalability. NewSQL attempts to have both. And analytical systems optimize for a different class of workloads. The field's history is not a linear progression but a branching tree, where each new branch addresses a limitation of the existing ones, and where the old branches continue to thrive alongside the new. The central questions—how to store, query, and update data reliably and efficiently—remain constant, but the answers are always contingent on the hardware, the workload, and the application's requirements.