Distributed Systems

A distributed system is a collection of independent computers that appears to its users as a single coherent system.

12 frameworks, from 1961 to today · Computer Systems · Computer Science

still a working view ended · each leads to its entry below

Overview

A distributed system is a collection of independent computers that appears to its users as a single coherent system1. That definition, in one form or another, has guided the field since the 1970s, and it captures the central tension: the machines are separate, so they can fail separately, communicate only by messages that arrive late or not at all, and have no shared clock—yet the software must behave as if they were one. What is at stake is scale and reliability. A single computer can be made fast only up to a point; a network of computers can grow without bound and survive the loss of individual members, but only if the software confronts the physics of partial failure. Distributed systems is the discipline of building that software.

Time-sharing and the first remote terminals

The prehistory lies in the 1960s, when computers were batch machines fed by decks of punched cards. Time-sharing changed that: a single computer switched rapidly among many users, each at a terminal, giving the illusion that the machine belonged to each of them alone2. Project MAC at MIT, running the Compatible Time-Sharing System from 1961, and the Dartmouth Time-Sharing System of 1964 were early landmarks. The terminals were remote—connected by telephone lines—so users were already interacting with a machine at a distance. The step to distributed systems came when researchers asked whether several computers could cooperate rather than merely serve terminals. ARPANET, whose first link was established in 1969, made that question practical by connecting machines at different sites.

Remote procedure calls and the client–server model

Early networked programs spoke in raw messages over sockets, and every application had to invent its own protocol. The idea of the remote procedure call (RPC), developed at Xerox PARC in the 1970s and published in a widely read 1984 paper by Andrew Birrell and Bruce Nelson, made a network call look like an ordinary function call: the programmer writes result = f(x) and a stub on the client marshals the arguments, sends them, and waits for the reply. This hid the network behind a familiar abstraction and gave rise to the client–server model, in which one program requests services and another provides them34. The abstraction is seductive but leaky—a local call either succeeds or crashes the program, while a remote call can hang forever if the reply is lost—and much later work on timeouts, retries, and idempotency grew from that gap.

Message passing and process algebra

Where RPC assumed request and reply, other systems were built around asynchronous messages. The actor model, proposed by Carl Hewitt in 1973, treated computation as independent actors that only communicate by sending messages, with no shared state. In parallel, a theoretical strand asked how to reason about concurrent processes at all. Tony Hoare's Communicating Sequential Processes (CSP), published in 1978, and Robin Milner's Calculus of Communicating Systems (CCS) from 1980 gave formal languages in which processes and their interactions could be described and compared56. These calculi mattered because they let designers prove properties of protocols before building them, and CSP's ideas later shaped the concurrency model of the Go programming language5.

Distributed file systems

If many machines must cooperate, they need a shared view of stored data. The Network File System (NFS), released by Sun Microsystems in 1984, let a client mount a directory on a remote server as if it were local7. The Andrew File System, developed at Carnegie Mellon in the 1980s, took a different route: it cached whole files on client machines and used callbacks to keep caches coherent, trading fine-grained consistency for scalability across thousands of workstations. These systems taught the field that caching is essential and that keeping caches consistent is where the difficulty hides.

Replication and consensus

The deepest problem is agreement. If data is copied onto several machines for availability, the copies must not diverge, and machines must agree on an order of operations even when some are unreachable. Leslie Lamport's 1978 paper on clocks and the ordering of events in a distributed system showed that physical clocks cannot establish a reliable global order, so logical clocks and timestamps must do the work. The state machine replication approach—run the same deterministic operations in the same order on every replica—reduced the problem to agreeing on a sequence. Lamport's Paxos, circulated in the late 1980s and published in 1998, became the canonical consensus protocol; it guarantees that a majority of replicas agree on a value despite crashes and message delays. Practical systems such as Google's Chubby lock service (described in 2006) and later Raft, designed by Diego Ongaro and John Ousterhout in 2014 for understandability, brought consensus into everyday infrastructure.

Middleware and service-oriented architecture

As networks grew, a layer of software called middleware emerged to connect heterogeneous machines and languages. The Open Software Foundation's Distributed Computing Environment (DCE) in the early 1990s bundled RPC, naming, and security. Object-oriented middleware followed—CORBA, standardized in 1991, and Microsoft's DCOM—letting programs call methods on remote objects. In the 2000s, service-oriented architecture (SOA) recast applications as collections of loosely coupled services communicating through standard web protocols such as SOAP and later REST. The recurring lesson was that loose coupling and standard interfaces buy interoperability at the cost of performance and complexity.

Grid computing

The 1990s and early 2000s saw grid computing, which aimed to pool computing power across institutions—often universities and research labs—the way an electrical grid pools generators. Projects such as Globus, begun in 1995, built toolkits for authentication, job scheduling, and data movement across administrative domains. Grids proved that large-scale resource sharing was possible, but they struggled with heterogeneity and with the absence of a single owner, problems that the commercial cloud would later address with centralized control.

CAP and consistency models

In 2000, Eric Brewer articulated what became the CAP theorem: a distributed data store cannot simultaneously guarantee consistency, availability, and partition tolerance8. Since network partitions are unavoidable, real systems must choose between strong consistency and availability during a partition. Seth Gilbert and Nancy Lynch gave a formal proof in 20028. CAP forced designers to state their consistency guarantees explicitly, and a spectrum of models—linearizability, sequential consistency, causal consistency—entered common vocabulary. The theorem is often oversimplified, but its practical effect was to make consistency a design decision rather than an afterthought.

Peer-to-peer systems and distributed hash tables

Around the same time, peer-to-peer file sharing showed that millions of ordinary machines could cooperate without any central server. Napster (1999) still relied on a central index; Gnutella (2000) removed it but flooded the network with queries. The structured answer was the distributed hash table: Chord, from MIT in 2001, and CAN and Pastry placed keys on a ring or space so that any node could locate any key in a logarithmic number of hops. These systems demonstrated self-organization at scale and influenced later content distribution and storage designs.

MapReduce and data-parallel batch processing

Google's MapReduce, described in a 2004 paper by Jeffrey Dean and Sanjay Ghemawat, took a different tack: rather than expose the network, it hid it behind two simple operations—map, which transforms records, and reduce, which aggregates them—and let the framework handle partitioning, scheduling, and recovery from failed workers. The open-source Hadoop implementation made the model widely available910. Its limitation was latency: batch jobs took minutes to hours, which motivated later stream-processing systems.

Cloud computing and data-center resource management

The 2006 launch of Amazon Web Services' Elastic Compute Cloud turned distributed systems into a utility: users rented virtual machines by the hour, and the provider's job became packing many tenants onto shared hardware. Google's papers on the Google File System (2003), Bigtable (2006), and later Borg (2015) described how a single company managed clusters of thousands of machines. Resource managers such as Apache Mesos (2011) and Kubernetes (2014) generalized scheduling and orchestration. The data center itself became the unit of design, and concerns shifted to energy, isolation, and automated failure recovery.

Eventual consistency and CRDTs

For geo-replicated services, waiting for consensus on every write is too slow. The Dynamo system at Amazon, described in 2007, embraced eventual consistency: replicas may diverge temporarily but converge if writes stop. The risk is conflicting updates, and conflict-free replicated data types (CRDTs), formalized by Marc Shapiro and colleagues around 2011, offered data structures—counters, sets, registers—whose operations commute so that replicas merge automatically without coordination11. These ideas underpin many modern collaborative applications and globally distributed databases.

The present

Today the field runs on a layered inheritance. Consensus protocols such as Paxos and Raft sit inside coordination services and replicated databases; RPC and its modern descendants (gRPC, REST) carry most inter-service traffic; orchestration platforms schedule containers across data centers; and consistency is chosen per application, from strong linearizability to CRDT-based eventual convergence. Open questions remain sharp: how to verify large distributed protocols, how to reason about failures that are rare but correlated, how to keep latency low across continents without weakening guarantees, and how to make the whole stack comprehensible to the engineers who must operate it. The machines are still separate, and the software still must pretend otherwise.

References

  1. Distributed computing (Introduction), Wikipedia
  2. Time-sharing, Wikipedia
  3. Remote procedure call, Wikipedia
  4. Client–server model, Wikipedia
  5. Communicating sequential processes, Wikipedia
  6. Process calculus (History), Wikipedia
  7. Network File System, Wikipedia
  8. CAP theorem (introduction, History), Wikipedia
  9. MapReduce, Wikipedia
  10. Apache Hadoop, Wikipedia
  11. Conflict-free replicated data type, Wikipedia

The frameworks of Distributed Systems

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1961-1975

    Time-sharing

    How can one computer serve many users at once?

    Fly to its tower in Nousfarer

  2. 1973-Present

    Message Passing and Process Algebra

    How do independent processes communicate and coordinate?

    Fly to its tower in Nousfarer

  3. 1975-Present

    Remote Procedure Call and Client–Server

    How can a program call code on another machine?

    Fly to its tower in Nousfarer

  4. 1978-Present

    Replication and Consensus

    How can copies of data stay in agreement?

    Fly to its tower in Nousfarer

  5. 1984-2005

    Distributed File Systems

    How can many machines share one view of files?

    Fly to its tower in Nousfarer

  6. 1991-2015

    Middleware and Service-Oriented Architecture

    How can heterogeneous systems be connected by common software?

    Fly to its tower in Nousfarer

  7. 1995-2010

    Grid Computing

    How can institutions pool computing resources across domains?

    Fly to its tower in Nousfarer

  8. 1999-2012

    Peer-to-Peer Systems and Distributed Hash Tables

    How can millions of peers cooperate without a central server?

    Fly to its tower in Nousfarer

  9. 2000-Present

    CAP and Consistency Models

    What must a distributed store give up during a partition?

    Fly to its tower in Nousfarer

  10. 2004-2015

    MapReduce

    How can simple operations process huge datasets reliably?

    Fly to its tower in Nousfarer

  11. 2006-Present

    Cloud computing

    How can computing be rented as a utility?

    Fly to its tower in Nousfarer

  12. 2007-Present

    Eventual Consistency and CRDTs

    How can replicas converge without waiting for agreement?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Time-sharingMessage Passing and Process AlgebraRemote Procedure Call and Client–Serv…Replication and ConsensusDistributed File SystemsMiddleware and Service-Oriented Archi…Grid ComputingPeer-to-Peer Systems and Distributed …CAP and Consistency ModelsMapReduceCloud computingEventual Consistency and CRDTs

Beyond the field

Where Distributed Systems's ideas came from in other fields, and what they led to there: the strongest 12 of 93 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Distributed Systems is a planet in Nousfarer

Its city has a tower for each of these 12 frameworks. Land, walk in, and climb one concept at a time.

Land on Distributed Systems

Related fields

Fields its ideas are linked to

More in Computer Systems

See where Distributed Systems sits on the Map of Human Knowledge