Big Data Processing

Big data processing is the set of techniques and systems built to store, move, and compute over datasets too large, too fast, or too unstructured for a single conventional database server.

12 frameworks, from 1983 to today · Data Science · Computer Science

still a working view ended · each leads to its entry below

Overview

Big data processing is the set of techniques and systems built to store, move, and compute over datasets too large, too fast, or too unstructured for a single conventional database server1. What is at stake is deceptively simple: as data outgrows one machine, how do you keep computing answers without losing correctness, speed, or your mind? The history of the field is a long argument between two instincts—bring the computation to the data across many cheap machines, or buy one enormous machine and keep everything close—and the gradual discovery that the right answer depends on the job.

Parallel Databases and the First "Big Data" Problem

The problem was already visible in the 1980s. Relational databases, built on Edgar Codd's 1970 model of tables manipulated by a declarative query language, worked well until the data and the query load outgrew a single processor. The response was the parallel database: split a table across many disks and processors, run the query on all of them at once, and combine the results2. Systems such as Teradata (commercial from the late 1970s), Tandem's NonStop SQL, and research prototypes like Gamma at the University of Wisconsin in the mid-1980s established the core ideas—data partitioning, parallel scans, and a shared-nothing architecture in which each node has its own processor, memory, and disk and communicates only by messages.

The key insight was that a declarative query lets the system decide how to execute it, so the optimizer can choose a parallel plan. A famous 1990 paper by David DeWitt and Jim Gray, "Parallel Database Systems: The Future of High Performance Database Systems," argued that this approach would dominate. For structured data and SQL workloads, it largely did. But parallel databases were expensive, sold as appliances, and assumed data was clean, schema-conforming, and loaded deliberately. The web would break all three assumptions.

Distributed File Systems and the Commodity Turn

By the late 1990s, web companies faced a new shape of problem: petabytes of logs, crawled pages, and clickstreams, growing continuously, stored on hardware that failed constantly. Two Google papers set the agenda. The 2003 paper on the Google File System described a design in which files are split into large chunks, each replicated across many commodity machines, with a single master tracking where chunks live. Cheap disks and routine failure became the design premise rather than an exception. The 2004 paper on MapReduce described a programming model for processing that data1.

MapReduce borrowed its two primitives from functional programming: a map step transforms each record independently, and a reduce step aggregates the results by key3. The system handled the hard parts—splitting input, scheduling work near the data, recovering from failed workers by re-running their tasks. Its famous demonstration was counting word frequencies across a huge corpus in a few lines of code. The appeal was that ordinary programmers could write parallel programs without thinking about parallelism.

Hadoop and the Open-Source Ecosystem

Google's papers were descriptions, not products. Doug Cutting and Mike Cafarella built open-source implementations: the Hadoop Distributed File System (named for Cutting's son's toy elephant) and an implementation of MapReduce, which became an Apache project in 2006 and was adopted by Yahoo and others. Around it grew an ecosystem: Hive, developed at Facebook from 2007, which let analysts write SQL-like queries translated into MapReduce jobs; Pig, a dataflow scripting language from Yahoo; HBase, a column-family store modeled on Google's Bigtable; and later Spark and Tez as alternative execution engines. The ecosystem's achievement was to make cluster computing available to any organization willing to assemble commodity hardware.

It also revealed costs. MapReduce wrote intermediate results to disk between every stage, which made iterative algorithms—machine learning above all—painfully slow. And the "one framework for everything" ambition proved overreach.

NoSQL, CAP, and the Consistency Trade-off

Meanwhile, web-scale services were straining relational databases on a different axis: many concurrent users, geographically distributed data, and availability requirements that could not tolerate a single master going down. A 2000 lecture by Eric Brewer, formalized later by Seth Gilbert and Nancy Lynch, became the CAP theorem: a distributed system facing a network partition must choose between consistency and availability4. In practice, builders of large internet systems chose availability and eventual consistency—replicas may diverge temporarily but converge if updates stop5.

The systems that followed were grouped, somewhat loosely, as NoSQL: key-value stores like Amazon's Dynamo (described in 2007) and Redis; document stores like MongoDB; wide-column stores like Bigtable and Cassandra; graph databases like Neo4j. They traded the relational model's fixed schema and joins for flexible data models and horizontal scale6. The label was always a bit of a marketing umbrella, and the trade-offs were real: giving up joins and transactions pushed complexity into application code.

In-Memory Computing and the Return of Generality

The next turning point came from a Berkeley research group7. Their 2010 paper on Spark argued that MapReduce's disk-based dataflow was the wrong abstraction for two growing workloads: iterative algorithms and interactive queries. Spark kept data in memory across operations as resilient distributed datasets—immutable collections that could be rebuilt from lineage if a node failed—and exposed a richer set of operators than map and reduce. The result was often orders of magnitude faster for machine learning and exploratory analysis, and Spark became the dominant general-purpose cluster engine through the 2010s. Its SQL layer, Spark SQL, and the DataFrame API also signaled a reconciliation: the declarative, optimizer-driven approach of the parallel database was coming back, now on commodity clusters.

Streams, NewSQL, and the Cloud

Three further threads reshaped the field. Stream processing addressed data that arrives continuously rather than in stored batches: systems like Apache Storm (2011), Apache Flink, and Kafka Streams compute over unbounded flows, often with windows and exactly-once guarantees, and the boundary between batch and streaming blurred as engines like Flink and Spark's structured streaming unified the two models.

NewSQL responded to a complaint about NoSQL: many applications wanted horizontal scale and transactions8. Systems such as Google Spanner (2012), which used synchronized clocks to provide globally consistent transactions, and CockroachDB and VoltDB pursued this, showing that the CAP trade-off could be softened with better engineering.

Cloud-native processing changed the economics. Services like Amazon EMR, Google BigQuery, and Snowflake separated storage from compute, letting users rent clusters by the hour or query without managing servers at all. Elasticity—scaling up for a job and down afterward—became the default expectation.

Data Lakes and the Lakehouse

As storage cheapened, organizations began dumping raw data of every format into centralized repositories—data lakes—on distributed file systems or object stores, with the idea that schema could be applied at read time. Many lakes became swamps: without transactions or metadata management, data quality and consistency suffered. The lakehouse architecture, articulated around 2020 and implemented in open table formats such as Delta Lake, Apache Iceberg, and Apache Hudi, added transactional guarantees, schema enforcement, and time travel on top of lake storage, aiming to give warehouse-grade reliability to cheap, open storage910. It is, in a sense, the parallel database's discipline returning to the data lake's economics.

Where the Field Stands

Today the main approaches coexist rather than compete: cloud data warehouses for structured analytics, lakehouse formats for mixed and open data, Spark and Flink for large-scale batch and stream computation, and specialized stores for particular access patterns. The recurring lesson of the history is that the pendulum swings between generality and specialization, and between moving computation to data and centralizing data near powerful compute. Open questions remain: how to govern and secure data spread across many systems; how to make streaming and batch truly one model; how to keep costs and energy use under control as clusters grow; and how to serve machine-learning workloads that need both massive throughput and low-latency iteration. The field began by asking how to compute over data too big for one machine, and it is still answering that question in new forms.

References

  1. Big data (introduction, Definition, Architecture), Wikipedia
  2. Parallel database, Wikipedia
  3. MapReduce, Wikipedia
  4. CAP theorem (introduction, History), Wikipedia
  5. Eventual consistency, Wikipedia
  6. NoSQL, Wikipedia
  7. Apache Spark, Wikipedia
  8. NewSQL, Wikipedia
  9. Lakehouse, Wikipedia
  10. Apache Iceberg (introduction, History), Wikipedia

The frameworks of Big Data Processing

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1983-Present
  2. 2000-Present

    CAP and Eventual Consistency

    What trade-offs must distributed systems make between consistency and availability?

    Fly to its tower in Nousfarer

  3. 2003-Present

    Distributed File Systems

    How can files be stored reliably across many unreliable machines?

    Fly to its tower in Nousfarer

  4. 2004-2015
  5. 2006-2016
  6. 2007-Present

    NoSQL

    How can databases scale horizontally without the relational model?

    Fly to its tower in Nousfarer

  7. 2010-Present

    Data Lake Architecture

    How can raw data of any format be stored centrally and queried later?

    Fly to its tower in Nousfarer

  8. 2010-Present
  9. 2011-Present
  10. 2012-Present
  11. 2012-Present

    NewSQL

    How can databases provide both horizontal scale and ACID transactions?

    Fly to its tower in Nousfarer

  12. 2020-Present

    Lakehouse

    How can data lakes gain the reliability of data warehouses?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Parallel databaseCAP and Eventual ConsistencyDistributed File SystemsMapReduceHadoop EcosystemNoSQLData Lake ArchitectureIn-Memory Cluster ComputingStream ProcessingCloud-Native Data ProcessingNewSQLLakehouse

Beyond the field

Where Big Data Processing's ideas came from in other fields, and what they led to there: the strongest 12 of 66 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Big Data Processing is a planet in Nousfarer

Its city has a tower for each of these 12 frameworks. Land, walk in, and climb one concept at a time.

Land on Big Data Processing

Related fields

Fields its ideas are linked to

More in Data Science

See where Big Data Processing sits on the Map of Human Knowledge