Loading field map
Data systems are the engineered infrastructure for collecting, storing, processing, and analyzing data at scale. They form the practical backbone of data science, transforming raw data into usable, reliable, and accessible resources. The subfield studies the design, implementation, and operation of these systems, addressing fundamental tensions between performance, consistency, availability, and complexity.
The core challenge of data systems is managing data across multiple machines and over time while preserving its integrity and usefulness. Key questions include:
The stakes are high. Flawed data systems can lead to lost revenue, incorrect decisions, privacy breaches, or catastrophic failures in critical infrastructure. Conversely, well-designed systems enable the large-scale analytics, machine learning, and real-time services that define modern technology.
Data systems emerged from the need to manage data beyond the capacity of simple file storage. Early precursors include punched-card tabulating machines and magnetic tape-based record systems, but these were not interactive or general-purpose. The modern subfield began with the development of database management systems (DBMS) in the 1960s and 1970s.
The relational model, proposed by Edgar F. Codd in 1970, was a watershed. It separated the logical structure of data (tables, rows, columns) from its physical storage, allowing users to query data declaratively using languages like SQL. This abstraction made databases far more flexible and easier to use than earlier hierarchical or network models. Relational database systems (e.g., IBM DB2, Oracle, later PostgreSQL and MySQL) became the dominant paradigm for business data processing for decades.
However, the rise of the internet and web-scale applications in the late 1990s and 2000s exposed limitations of traditional relational databases. Companies like Google, Amazon, and Facebook needed to store and process petabytes of data across thousands of machines, with requirements for high availability and low latency that strained the consistency guarantees of relational systems. This led to the development of "NoSQL" systems (e.g., Google's Bigtable, Amazon's Dynamo, Apache Cassandra, MongoDB), which relaxed some of the relational model's constraints to achieve scalability and availability.
The 2010s saw a maturation and consolidation of ideas. The "NewSQL" movement attempted to combine the scalability of NoSQL with the strong consistency and SQL interface of relational databases (e.g., Google Spanner, CockroachDB). Meanwhile, systems for batch processing (e.g., Apache Hadoop MapReduce, Apache Spark) and stream processing (e.g., Apache Kafka, Apache Flink) emerged to handle analytical workloads on large datasets. The field is now characterized by a diverse ecosystem of specialized systems, each optimized for different trade-offs.
Data systems are not a single monolithic technology but a landscape of approaches that address different parts of the problem space. These approaches coexist and often complement each other.
The relational database, with its structured schema and SQL query language, remains the workhorse of transactional data processing. Its organizing principle is the ACID set of guarantees: Atomicity (transactions complete fully or not at all), Consistency (transactions bring the database from one valid state to another), Isolation (concurrent transactions appear to run sequentially), and Durability (committed data survives failures).
The problem ACID databases address is maintaining data integrity in the face of concurrent access and failures. They assume that correctness is paramount and that the system can afford to wait for consensus or reject operations that would violate constraints. Their limits become apparent at extreme scale: enforcing strong consistency across geographically distributed replicas introduces latency and reduces availability during network partitions. For this reason, ACID databases are best suited for applications where data accuracy is critical (e.g., financial ledgers, inventory management) and where the workload fits on a single server or a small cluster.
NoSQL systems emerged to handle workloads that relational databases could not serve efficiently: massive scale, flexible schemas, or high-velocity writes. They are often understood through the lens of the CAP theorem, which states that a distributed data system can guarantee at most two of three properties: Consistency (all nodes see the same data at the same time), Availability (every request receives a non-error response), and Partition Tolerance (the system continues to operate despite network failures).
NoSQL systems typically prioritize availability and partition tolerance over strong consistency, accepting "eventual consistency" where data updates propagate gradually. They are organized into several families:
The problem NoSQL systems address is scalability and operational simplicity for specific use cases. Their limits include weaker consistency guarantees, which can complicate application logic, and a lack of a universal query language, which can increase development effort. They are not a replacement for relational databases but a complementary tool for workloads where the relational model's constraints are too costly.
A separate axis of data systems concerns the timing of processing. Batch processing systems (e.g., Hadoop MapReduce, Spark) operate on data at rest, processing large volumes in periodic jobs. Stream processing systems (e.g., Apache Kafka Streams, Apache Flink, Apache Storm) operate on data in motion, processing events as they arrive with low latency.
The problem stream processing addresses is the need for real-time insights and actions—fraud detection, monitoring, live dashboards—where batch processing's delay is unacceptable. Its limits include the difficulty of exactly-once processing semantics and the complexity of managing state across a distributed stream.
The Lambda architecture attempted to combine both by running batch and stream layers in parallel, merging their results. The Kappa architecture simplified this by treating all data as a stream, using a replayable log (e.g., Kafka) to handle both real-time and historical processing. The Kappa architecture has gained traction as stream processing engines have matured, but batch processing remains important for workloads that benefit from full data scans and complex joins.
A fundamental tension in distributed data systems is how to achieve agreement among nodes. Distributed consensus algorithms (e.g., Paxos, Raft) allow a group of machines to agree on a value even if some fail. These algorithms are the foundation for strongly consistent systems like Google Spanner, etcd, and ZooKeeper.
The problem consensus addresses is ensuring that all nodes in a distributed system have a consistent view of the data, even under failures. Its limits are performance: consensus requires multiple rounds of communication, which adds latency and reduces throughput. Systems that use consensus are typically slower than those that accept eventual consistency, but they provide guarantees that simplify application development.
As data volumes grew, organizations began storing raw data in its native format in data lakes (e.g., on Amazon S3, HDFS) rather than loading it into structured databases. This approach offers flexibility and low storage cost but can lead to data quality and governance problems (a "data swamp").
The lakehouse architecture (e.g., Databricks Delta Lake, Apache Iceberg) attempts to combine the flexibility of a data lake with the reliability and performance of a data warehouse. It adds ACID transactions, schema enforcement, and indexing on top of cloud object storage. The problem lakehouses address is the need for a single platform that supports both data engineering and data science workloads without duplicating data. Their limits include the complexity of managing metadata at scale and the performance overhead of enforcing ACID on object storage.
The current landscape of data systems is characterized by pluralism and convergence. No single approach dominates; instead, organizations deploy multiple systems for different purposes, often connected by data pipelines. Key trends include:
The field remains active, with ongoing research into new consistency models, hardware-accelerated processing (e.g., GPUs, FPGAs, persistent memory), and systems that can handle the scale and diversity of data from the Internet of Things, machine learning pipelines, and real-time analytics. The central challenge—building reliable, performant, and usable systems for data at scale—continues to drive innovation.