- Atlas
- Computer Science
- Big Data Processing
Big Data Processing
Big data processing is the set of techniques and systems built to store, move, and compute over datasets too large, too fast, or too unstructured for a single conventional database server.
12 frameworks, from 1983 to today · Data Science · Computer Science
- Parallel database 1983-Present
- CAP and Eventual Consistency 2000-Present
- Distributed File Systems 2003-Present
- MapReduce 2004-2015
- Hadoop Ecosystem 2006-2016
- NoSQL 2007-Present
- Data Lake Architecture 2010-Present
- In-Memory Cluster Computing 2010-Present
- Stream Processing 2011-Present
- Cloud-Native Data Processing 2012-Present
- NewSQL 2012-Present
- Lakehouse 2020-Present
Overview
Big data processing is the set of techniques and systems built to store, move, and compute over datasets too large, too fast, or too unstructured for a single conventional database server1. What is at stake is deceptively simple: as data outgrows one machine, how do you keep computing answers without losing correctness, speed, or your mind? The history of the field is a long argument between two instincts—bring the computation to the data across many cheap machines, or buy one enormous machine and keep everything close—and the gradual discovery that the right answer depends on the job.
Parallel Databases and the First "Big Data" Problem
The problem was already visible in the 1980s. Relational databases, built on Edgar Codd's 1970 model of tables manipulated by a declarative query language, worked well until the data and the query load outgrew a single processor. The response was the parallel database: split a table across many disks and processors, run the query on all of them at once, and combine the results2. Systems such as Teradata (commercial from the late 1970s), Tandem's NonStop SQL, and research prototypes like Gamma at the University of Wisconsin in the mid-1980s established the core ideas—data partitioning, parallel scans, and a shared-nothing architecture in which each node has its own processor, memory, and disk and communicates only by messages.
The key insight was that a declarative query lets the system decide how to execute it, so the optimizer can choose a parallel plan. A famous 1990 paper by David DeWitt and Jim Gray, "Parallel Database Systems: The Future of High Performance Database Systems," argued that this approach would dominate. For structured data and SQL workloads, it largely did. But parallel databases were expensive, sold as appliances, and assumed data was clean, schema-conforming, and loaded deliberately. The web would break all three assumptions.
Distributed File Systems and the Commodity Turn
By the late 1990s, web companies faced a new shape of problem: petabytes of logs, crawled pages, and clickstreams, growing continuously, stored on hardware that failed constantly. Two Google papers set the agenda. The 2003 paper on the Google File System described a design in which files are split into large chunks, each replicated across many commodity machines, with a single master tracking where chunks live. Cheap disks and routine failure became the design premise rather than an exception. The 2004 paper on MapReduce described a programming model for processing that data1.
MapReduce borrowed its two primitives from functional programming: a map step transforms each record independently, and a reduce step aggregates the results by key3. The system handled the hard parts—splitting input, scheduling work near the data, recovering from failed workers by re-running their tasks. Its famous demonstration was counting word frequencies across a huge corpus in a few lines of code. The appeal was that ordinary programmers could write parallel programs without thinking about parallelism.
Hadoop and the Open-Source Ecosystem
Google's papers were descriptions, not products. Doug Cutting and Mike Cafarella built open-source implementations: the Hadoop Distributed File System (named for Cutting's son's toy elephant) and an implementation of MapReduce, which became an Apache project in 2006 and was adopted by Yahoo and others. Around it grew an ecosystem: Hive, developed at Facebook from 2007, which let analysts write SQL-like queries translated into MapReduce jobs; Pig, a dataflow scripting language from Yahoo; HBase, a column-family store modeled on Google's Bigtable; and later Spark and Tez as alternative execution engines. The ecosystem's achievement was to make cluster computing available to any organization willing to assemble commodity hardware.
It also revealed costs. MapReduce wrote intermediate results to disk between every stage, which made iterative algorithms—machine learning above all—painfully slow. And the "one framework for everything" ambition proved overreach.
NoSQL, CAP, and the Consistency Trade-off
Meanwhile, web-scale services were straining relational databases on a different axis: many concurrent users, geographically distributed data, and availability requirements that could not tolerate a single master going down. A 2000 lecture by Eric Brewer, formalized later by Seth Gilbert and Nancy Lynch, became the CAP theorem: a distributed system facing a network partition must choose between consistency and availability4. In practice, builders of large internet systems chose availability and eventual consistency—replicas may diverge temporarily but converge if updates stop5.
The systems that followed were grouped, somewhat loosely, as NoSQL: key-value stores like Amazon's Dynamo (described in 2007) and Redis; document stores like MongoDB; wide-column stores like Bigtable and Cassandra; graph databases like Neo4j. They traded the relational model's fixed schema and joins for flexible data models and horizontal scale6. The label was always a bit of a marketing umbrella, and the trade-offs were real: giving up joins and transactions pushed complexity into application code.
In-Memory Computing and the Return of Generality
The next turning point came from a Berkeley research group7. Their 2010 paper on Spark argued that MapReduce's disk-based dataflow was the wrong abstraction for two growing workloads: iterative algorithms and interactive queries. Spark kept data in memory across operations as resilient distributed datasets—immutable collections that could be rebuilt from lineage if a node failed—and exposed a richer set of operators than map and reduce. The result was often orders of magnitude faster for machine learning and exploratory analysis, and Spark became the dominant general-purpose cluster engine through the 2010s. Its SQL layer, Spark SQL, and the DataFrame API also signaled a reconciliation: the declarative, optimizer-driven approach of the parallel database was coming back, now on commodity clusters.
Streams, NewSQL, and the Cloud
Three further threads reshaped the field. Stream processing addressed data that arrives continuously rather than in stored batches: systems like Apache Storm (2011), Apache Flink, and Kafka Streams compute over unbounded flows, often with windows and exactly-once guarantees, and the boundary between batch and streaming blurred as engines like Flink and Spark's structured streaming unified the two models.
NewSQL responded to a complaint about NoSQL: many applications wanted horizontal scale and transactions8. Systems such as Google Spanner (2012), which used synchronized clocks to provide globally consistent transactions, and CockroachDB and VoltDB pursued this, showing that the CAP trade-off could be softened with better engineering.
Cloud-native processing changed the economics. Services like Amazon EMR, Google BigQuery, and Snowflake separated storage from compute, letting users rent clusters by the hour or query without managing servers at all. Elasticity—scaling up for a job and down afterward—became the default expectation.
Data Lakes and the Lakehouse
As storage cheapened, organizations began dumping raw data of every format into centralized repositories—data lakes—on distributed file systems or object stores, with the idea that schema could be applied at read time. Many lakes became swamps: without transactions or metadata management, data quality and consistency suffered. The lakehouse architecture, articulated around 2020 and implemented in open table formats such as Delta Lake, Apache Iceberg, and Apache Hudi, added transactional guarantees, schema enforcement, and time travel on top of lake storage, aiming to give warehouse-grade reliability to cheap, open storage910. It is, in a sense, the parallel database's discipline returning to the data lake's economics.
Where the Field Stands
Today the main approaches coexist rather than compete: cloud data warehouses for structured analytics, lakehouse formats for mixed and open data, Spark and Flink for large-scale batch and stream computation, and specialized stores for particular access patterns. The recurring lesson of the history is that the pendulum swings between generality and specialization, and between moving computation to data and centralizing data near powerful compute. Open questions remain: how to govern and secure data spread across many systems; how to make streaming and batch truly one model; how to keep costs and energy use under control as clusters grow; and how to serve machine-learning workloads that need both massive throughput and low-latency iteration. The field began by asking how to compute over data too big for one machine, and it is still answering that question in new forms.
References
- Big data (introduction, Definition, Architecture), Wikipedia
- Parallel database, Wikipedia
- MapReduce, Wikipedia
- CAP theorem (introduction, History), Wikipedia
- Eventual consistency, Wikipedia
- NoSQL, Wikipedia
- Apache Spark, Wikipedia
- NewSQL, Wikipedia
- Lakehouse, Wikipedia
- Apache Iceberg (introduction, History), Wikipedia
The frameworks of Big Data Processing
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1983-Present
- 2000-Present
CAP and Eventual Consistency
What trade-offs must distributed systems make between consistency and availability?
- 2003-Present
Distributed File Systems
How can files be stored reliably across many unreliable machines?
- 2004-2015
- 2006-2016
- 2007-Present
NoSQL
How can databases scale horizontally without the relational model?
- 2010-Present
Data Lake Architecture
How can raw data of any format be stored centrally and queried later?
- 2010-Present
- 2011-Present
- 2012-Present
- 2012-Present
NewSQL
How can databases provide both horizontal scale and ACID transactions?
- 2020-Present
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
NoSQL influenced by CAP and Eventual Consistency
The CAP theorem and the concept of eventual consistency provided the theoretical justification for NoSQL systems, which chose availability over strong consistency in the face of network partitions.
Carried by Eric Brewer; Seth Gilbert; Nancy Lynch.
MapReduce influenced by Distributed File Systems
MapReduce was designed to run on the Google File System, using its chunked, replicated storage to schedule computation near the data and recover from failures.
Carried by Jeffrey Dean; Sanjay Ghemawat; MapReduce: Simplified Data Processing on Large Clusters (2004).
Hadoop Ecosystem grew out of Distributed File Systems
The Hadoop Distributed File System was an open-source implementation of the Google File System, and it became the storage layer of the Hadoop ecosystem.
Carried by Doug Cutting; Mike Cafarella.
Hadoop Ecosystem grew out of MapReduce
Hadoop's MapReduce was a direct open-source implementation of Google's MapReduce programming model, and it served as the core execution engine of the early Hadoop ecosystem.
Carried by Doug Cutting; Mike Cafarella.
In-Memory Cluster Computing reacted against MapReduce
Spark was created as a reaction against MapReduce's disk-based dataflow, which was slow for iterative algorithms; it kept data in memory and exposed a richer set of operators.
Carried by Matei Zaharia; Resilient Distributed Datasets (2010).
In-Memory Cluster Computing replaced Hadoop Ecosystem
Spark gradually replaced Hadoop MapReduce as the dominant execution engine in the Hadoop ecosystem, offering faster in-memory computation and a more general API.
Carried by Matei Zaharia; Apache Spark.
Lakehouse grew out of Data Lake Architecture
The lakehouse architecture grew directly out of data lakes, adding transactional guarantees, schema enforcement, and metadata management to address the problem of data lakes becoming unreliable 'swamps'.
Carried by Delta Lake (2019); Apache Iceberg; Apache Hudi.
MapReduce influenced by Parallel database
The MapReduce paper's authors were aware of parallel database systems and the shared-nothing architecture; the design of scheduling work near data and partitioning input echoes parallel database techniques, though MapReduce rejected the declarative query model.
Carried by Jeffrey Dean; Sanjay Ghemawat; MapReduce: Simplified Data Processing on Large Clusters (2004).
In-Memory Cluster Computing influenced by Parallel database
Spark's creators argued that MapReduce's disk-based dataflow was inferior and explicitly brought back the declarative, optimizer-driven approach of parallel databases, as seen in Spark SQL and the DataFrame API.
Carried by Matei Zaharia; Resilient Distributed Datasets (2010).
NewSQL influenced by Parallel database
NewSQL systems such as Spanner and VoltDB adopted the shared-nothing architecture and partitioning techniques from parallel databases while adding horizontal scale and distributed transactions.
Carried by Spanner: Google's Globally-Distributed Database (2012).
Stream Processing influenced by MapReduce
Stream processing systems like Apache Storm and later Flink adopted MapReduce's model of distributed computation over partitioned data, extending it to unbounded streams.
Carried by Apache Storm; Apache Flink.
Data Lake Architecture influenced by Hadoop Ecosystem
The Hadoop ecosystem, particularly HDFS and Hive, provided the storage and query infrastructure that made early data lakes possible, allowing raw data to be stored cheaply and queried later.
Carried by Hadoop Distributed File System; Apache Hive.
NewSQL reacted against NoSQL
NewSQL emerged as a response to NoSQL's trade-offs, particularly the loss of transactions and joins, aiming to provide horizontal scalability with ACID guarantees.
Carried by Spanner: Google's Globally-Distributed Database (2012); VoltDB.
Stream Processing influenced by In-Memory Cluster Computing
Spark's structured streaming extended the in-memory cluster computing model to continuous data, unifying batch and stream processing under a single engine.
Carried by Apache Spark Structured Streaming.
Cloud-Native Data Processing influenced by In-Memory Cluster Computing
Cloud-native data processing services like Amazon EMR and Google BigQuery adopted the cluster computing model of Spark and Hadoop but offered it as a managed, elastic service.
Carried by Amazon EMR.
Lakehouse influenced by Cloud-Native Data Processing
The lakehouse architecture was built for cloud object storage and separated storage from compute, following the cloud-native model; it added transactional layers on top of cloud storage.
Carried by Delta Lake (2019); Apache Iceberg.
Cloud-Native Data Processing influenced by Stream Processing
Cloud providers began offering managed stream processing services (e.g., Google Cloud Dataflow, Amazon Kinesis), bringing stream processing into the cloud-native paradigm.
Carried by Google Cloud Dataflow; Amazon Kinesis.
Beyond the field
Where Big Data Processing's ideas came from in other fields, and what they led to there: the strongest 12 of 66 links.
Hadoop Ecosystem grew out of MapReduce in Data Science
The Hadoop ecosystem grew directly out of MapReduce: Hadoop implemented the MapReduce model as an open-source framework, with its MapReduce engine as the core component.
Carried by Doug Cutting.
Hadoop Ecosystem grew out of MapReduce in Distributed Systems
Hadoop's distributed processing engine is a direct open-source implementation of the MapReduce model described by Google in 2004.
Carried by Doug Cutting.
Hadoop Ecosystem grew out of MapReduce in ML Systems
The Hadoop ecosystem's processing layer is a direct implementation of the MapReduce model, the same idea filed under ML systems.
Carried by Doug Cutting.
Hadoop Ecosystem led to Big Data Platforms in Business Analytics
The Hadoop ecosystem is the open-source implementation lineage that big data platforms built on, with HDFS and MapReduce as the substrate for later platform architectures.
Carried by Doug Cutting.
NoSQL provoked NewSQL in Nosql Systems
NewSQL defined itself against the NoSQL movement's abandonment of ACID transactions and SQL, arguing that scalability and transactional guarantees could be combined. The term was coined in 2011 precisely to mark this rejection of NoSQL's trade-offs.
Carried by VoltDB; Matthew Aslett.
NoSQL influenced Polyglot persistence in Nosql Systems
The practice of combining multiple specialized stores arose from the spread of NoSQL systems in large-scale data processing, where different workloads were assigned to different engines. Polyglot persistence names that resulting architecture.
NoSQL provoked NewSQL in Distributed Databases
NewSQL arose as a reaction to NoSQL's abandonment of SQL and ACID, arguing that horizontal scalability could be had while keeping relational semantics and transactions; the NewSQL label was coined explicitly against the NoSQL movement.
Carried by Matthew Aslett; VoltDB.
Data Lake Architecture led to Lakehouse in Data Science
The lakehouse was proposed as a refinement of the data lake architecture, keeping low-cost open storage while adding schema, transactions and warehouse-style management. The Databricks team that advanced the lakehouse framed it explicitly as fixing the data lake's lack of ACID guarantees and governance.
Carried by Databricks; Michael Armbrust.
In-Memory Cluster Computing influenced by Cluster computing in Parallel Computing
In-memory cluster computing, as in Spark, was built directly on cluster computing infrastructure and its scheduling and fault-tolerance models, adding memory-resident datasets on top.
Carried by Apache Spark.
NewSQL influenced by Distributed transaction in Distributed Databases
NewSQL systems build on decades of distributed transaction research, adopting two-phase commit and consensus-based replication to give ACID guarantees across nodes.
Lakehouse influenced by Column-oriented and lakehouse analytics in Databases
The lakehouse architecture was built directly on columnar file formats and open table layers developed in column-oriented analytics, which supply its storage and metadata foundation.
Carried by Apache Parquet; Delta Lake.
Parallel database influenced MapReduce in Data Science
MapReduce borrowed techniques from parallel database systems, notably the use of partitioning, sorting and hash-based aggregation, and its authors explicitly compared it with parallel DBMS execution strategies.

Big Data Processing is a planet in Nousfarer
Its city has a tower for each of these 12 frameworks. Land, walk in, and climb one concept at a time.
Land on Big Data ProcessingRelated fields
Fields its ideas are linked to
- Databases 10 links
- Data Science 7 links
- Nosql Systems 6 links
- Distributed Systems 6 links
- Distributed Databases 5 links
- Computer Systems 5 links
- Parallel Computing 4 links
- Data Modeling 4 links
- Query Processing 4 links
- Cloud Computing 4 links
- Business Analytics 3 links
- Parallel Algorithms 2 links
More in Data Science
See where Big Data Processing sits on the Map of Human Knowledge