Reliability Fault Tolerance

Computers are built from parts that fail.

12 frameworks, from 1950 to today · Computer Systems · Computer Science

still a working view ended · each leads to its entry below

Overview

Computers are built from parts that fail1. A disk head crashes, a memory cell flips a bit under cosmic radiation, a power supply browns out, a network cable is cut, a program has a bug that only appears under load. Reliability and fault tolerance is the branch of computer systems that asks how to build machines and software that keep giving correct answers anyway—or, when they cannot, fail in a way that is predictable and contained1. The field's central move is to trade something (space, time, redundancy, money) for the probability of correctness1. Its history is the story of learning where that trade is worth making, and of discovering that the hardest failures are not the ones in hardware but the ones nobody anticipated.

Error-Correcting Codes and the Mathematical Foundation

The field begins with information theory. Claude Shannon's 1948 paper "A Mathematical Theory of Communication" established that a noisy channel has a capacity below which reliable communication is possible, and it framed the problem of encoding messages so that errors can be detected and corrected. Richard Hamming, working at Bell Labs and frustrated by punched-card machines that halted on a single parity error, published the Hamming code in 1950: by placing several parity bits at overlapping positions, a machine could not only detect that a bit had flipped but identify which one and fix it2. This was the first practical demonstration that redundancy could be structured—that a few extra bits bought a guaranteed correction capability rather than just a warning.

The decades that followed produced a family of codes that still underpin storage and communication: Reed–Solomon codes (1960), used in CDs, DVDs, QR codes, and deep-space telemetry; Bose–Chaudhuri–Hocquenghem (BCH) codes; and later convolutional and turbo codes. The key idea for systems builders is that error-correcting codes let a subsystem absorb faults locally, without the rest of the machine knowing. A memory with error-correcting code (ECC) silently corrects single-bit flips; a disk with Reed–Solomon parity reconstructs a lost sector. This local absorption is cheap and fast, and it remains the first line of defense in nearly every system today.

Reliability Engineering and the Statistics of Failure

In parallel, reliability engineering grew out of mid-century aerospace and electronics work. The question it asked was quantitative: given components with known failure rates, what is the probability a whole system survives a given mission? The bathtub curve—high infant mortality, a long flat useful life, then wear-out—became the standard mental model. The 1950s and 1960s saw the formalization of mean time between failures (MTBF), reliability block diagrams, and fault tree analysis, much of it driven by military and space programs where a failed component meant a lost mission. For computer systems, this tradition supplied the vocabulary of availability (the fraction of time a system is usable) and the discipline of reasoning about failure rates rather than just fixing bugs after the fact13.

Redundancy, Voting, and the Birth of Fault-Tolerant Computing

The idea that made fault tolerance a distinct engineering discipline was triple modular redundancy (TMR): run three copies of a computation and take a majority vote45. A single failure is outvoted; the system continues45. John von Neumann analyzed reliable computation from unreliable components in a 1956 paper, showing in principle that arbitrarily reliable machines could be built from unreliable parts given enough redundancy. The practical push came from a specific customer: the space program and, later, telephone switching. In the 1960s, IBM's System/360 Model 75 and the Saturn V launch computers incorporated redundancy; the Apollo Guidance Computer used a mix of hardware and software recovery.

The decisive institutional moment was the Jet Propulsion Laboratory's STAR (Self-Testing And Repairing) computer, designed in the late 1960s, and then the founding of the IEEE Technical Committee on Fault-Tolerant Computing in 1969, with the first International Symposium on Fault-Tolerant Computing (FTCS) held in 1971. Around the same time, the French company Thomson-CSF built the C.vmp and later the Intel 432 and Tandem systems pursued different routes. Tandem's NonStop line (1976) is the landmark commercial case: pairs of processors running in lockstep, comparing outputs, with a failover path so that a single hardware fault caused no interruption to transaction processing. The lesson Tandem taught the industry was that fault tolerance could be a product feature sold to banks and airlines, not just a research curiosity.

Checkpointing, Recovery, and the Software Side

Hardware redundancy is expensive1. A cheaper strategy is to let the system fail and then restart it from a saved state6. Checkpointing—periodically writing a process's memory and registers to stable storage—and rollback recovery—restoring that state after a failure—became a major research thread in the 1970s and 1980s. The problem is subtle: in a distributed system, processes must coordinate their checkpoints so that a consistent global state can be recovered, or they must log messages so that a rollback does not produce an impossible history. Work by K. Mani Chandy and Leslie Lamport on distributed snapshots (1985) and by David Johnson and Willy Zwaenepoel on pessimistic and optimistic message logging gave the field its standard techniques. Checkpointing remains the backbone of high-performance computing, where a job running for days on thousands of nodes must survive the loss of any one of them.

Transactions and the ACID Guarantee

A parallel and arguably more influential line of work came from databases. The transaction—a group of operations that either all happen or none do—with the properties atomicity, consistency, isolation, and durability (ACID) became the standard contract for reliable data management78. Jim Gray's work at IBM in the 1970s on transaction processing, write-ahead logging, and two-phase commit gave the field its core mechanisms. The insight was that reliability could be expressed as a programming model: the application programmer writes as if failures do not happen, and the system underneath handles the recovery. This abstraction—failures are hidden, not exposed—proved enormously productive and spread from databases into file systems (journaling), message queues, and eventually distributed systems.

Replicated State Machines and Consensus

If one machine can fail, run several and keep them in agreement9. The replicated state machine approach, formalized by Fred Schneider in 1990, says: if all replicas start in the same state and receive the same commands in the same order, they stay identical, and a client can tolerate the failure of a minority. The hard part is agreeing on the order—the consensus problem1011. Leslie Lamport's Paxos algorithm, circulated in the late 1980s and published in 1998, became the canonical solution, though its reputation for difficulty is legendary. The practical breakthrough came with Diego Ongaro and John Ousterhout's Raft (2014), designed explicitly for understandability, which spread rapidly through industry. Today consensus protocols underpin virtually every distributed database and coordination service.

End-to-End Reliability and the Argument Against In-Network Guarantees

A countercurrent ran through all of this. In 1984, Jerome Saltzer, David Reed, and David Clark published "End-to-End Arguments in System Design," arguing that reliability functions—error checking, delivery confirmation—are best placed at the endpoints of a communication, not in the network itself. The network should be simple; the endpoints should be smart12. This principle shaped the Internet's design and, more broadly, taught systems builders to ask where a reliability mechanism belongs13. It also set up a long-running tension: local mechanisms (ECC, checksums, retries) are cheap but can mask faults; end-to-end checks catch faults that local mechanisms miss. Modern systems use both, layered.

RAID and the Economics of Redundant Storage

In 1988, David Patterson, Garth Gibson, and Randy Katz at Berkeley published "A Case for Redundant Arrays of Inexpensive Disks" (RAID), showing that many cheap disks with parity could match the reliability of expensive ones at a fraction of the cost. RAID levels 1 through 5—mirroring, striping with parity, and their variants—became standard, and the paper's title (later softened to "independent") captured a shift in thinking: reliability was not only a property of individual components but of how they were arranged. RAID made the trade-off explicit and quantitative, and it remains the default way storage systems survive disk failure.

CAP, Consistency, and the Distributed Trade-off

In 2000, Eric Brewer articulated what became the CAP theorem—later proved by Seth Gilbert and Nancy Lynch—that a distributed system cannot simultaneously guarantee consistency, availability, and partition tolerance. The practical reading, popularized in the 2000s, was that under network partition you must choose between answering correctly and answering at all. This reframed reliability as a design choice rather than an absolute: different applications need different points on the spectrum14. The ensuing debate over "eventual consistency" and the rise of systems like Amazon's Dynamo (2007) pushed the field toward explicit, documented trade-offs rather than the assumption that ACID was always the goal.

Software Fault Tolerance at Internet Scale and Chaos Engineering

As systems grew to thousands of machines, the dominant failure mode shifted from component failure to emergent failure: cascading timeouts, retry storms, configuration drift, partial failures that no single component reports. The response was a set of software-level patterns—circuit breakers, bulkheads, graceful degradation, idempotent retries—codified in practices like Netflix's Hystrix and the broader discipline of site reliability engineering (SRE), named in a 2016 Google book. The most distinctive recent development is chaos engineering: deliberately injecting failures (killing instances, adding latency, partitioning networks) in production to discover weaknesses before they cause outages1516. Netflix's Chaos Monkey (2011) popularized the idea, and it has become a standard practice. Its logic descends directly from the field's oldest insight—that you cannot trust a system's reliability until you have tested it under the conditions it will actually face—but applies it to systems too complex to analyze on paper.

The Present

Today the field is layered. At the bottom, error-correcting codes and hardware redundancy absorb the common faults217. Above that, replication and consensus protocols tolerate whole-machine failures9. At the application level, transactions and idempotent retries handle the messy partial failures that distributed systems produce. And across all layers, chaos engineering and observability practices test whether the layers actually work together. The open questions are the ones the field has always faced in new form: how to reason about failures that emerge from interaction rather than from any single component; how to make formal verification of fault-tolerant protocols practical; and how to keep reliability affordable as systems scale beyond what any human can hold in mind. The history suggests the answer will again be a trade—some new structure of redundancy, placed where the last generation did not think to look.

References

  1. Fault tolerance (introduction, Criteria, Requirements, Disadvantages), Wikipedia
  2. Hamming code (introduction, History), Wikipedia
  3. Reliability engineering, Wikipedia
  4. Triple modular redundancy, Wikipedia
  5. Redundancy (engineering), Wikipedia
  6. Application checkpointing, Wikipedia
  7. ACID, Wikipedia
  8. Database transaction, Wikipedia
  9. State machine replication, Wikipedia
  10. Consensus (computer science), Wikipedia
  11. Paxos (computer science), Wikipedia
  12. End-to-end principle, Wikipedia
  13. Reliability (computer networking) (History), Wikipedia
  14. CAP theorem, Wikipedia
  15. Chaos engineering, Wikipedia
  16. Fault injection, Wikipedia
  17. ECC memory, Wikipedia

The frameworks of Reliability Fault Tolerance

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1950-Present

    Error detection and correction

    How can errors be detected and corrected?

    Fly to its tower in Nousfarer

  2. 1950-Present

    Reliability Engineering

    What is the probability a system survives?

    Fly to its tower in Nousfarer

  3. 1956-Present

    Hardware Redundancy and Voting

    How can unreliable parts compute reliably?

    Fly to its tower in Nousfarer

  4. 1965-Present

    Fault-Tolerant Computing

    How do computers keep working when parts fail?

    Fly to its tower in Nousfarer

  5. 1970-Present

    Application checkpointing

    How can a failed computation resume from saved state?

    Fly to its tower in Nousfarer

  6. 1970-Present

    Transactions and ACID

    How can data operations all happen or none?

    Fly to its tower in Nousfarer

  7. 1984-Present

    End-to-end principle

    Where should reliability functions be placed?

    Fly to its tower in Nousfarer

  8. 1988-Present

    RAID

    How can cheap disks match expensive reliability?

    Fly to its tower in Nousfarer

  9. 1990-Present

    State machine replication

    How do replicas agree on the same order?

    Fly to its tower in Nousfarer

  10. 2000-Present

    CAP and Consistency Trade-offs

    What must be traded off under network partition?

    Fly to its tower in Nousfarer

  11. 2010-Present

    Software Fault Tolerance at Internet Scale

    How can large services tolerate constant failure?

    Fly to its tower in Nousfarer

  12. 2011-Present

    Chaos Engineering

    How can we test systems by breaking them?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Error detection and correctionReliability EngineeringHardware Redundancy and VotingFault-Tolerant ComputingApplication checkpointingTransactions and ACIDEnd-to-end principleRAIDState machine replicationCAP and Consistency Trade-offsSoftware Fault Tolerance at Internet …Chaos Engineering

Beyond the field

Where Reliability Fault Tolerance's ideas came from in other fields, and what they led to there: the strongest 12 of 57 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Reliability Fault Tolerance is a planet in Nousfarer

Its city has a tower for each of these 12 frameworks. Land, walk in, and climb one concept at a time.

Land on Reliability Fault Tolerance

Related fields

Fields its ideas are linked to

More in Computer Systems

See where Reliability Fault Tolerance sits on the Map of Human Knowledge