Loading field map
Reliability fault tolerance is the engineering discipline concerned with designing computer systems that continue to deliver correct service despite the occurrence of faults. A fault is any defect, error, or malfunction—a bad memory cell, a software bug, a cosmic ray flipping a bit, a network cable cut, a power surge—that can cause a system to deviate from its specified behavior. When a fault becomes active, it produces an error, an incorrect internal state; if that error reaches the system's output, it becomes a failure, an observable deviation from what the user expects.
The central question of the field is deceptively simple: how can a system built from unreliable components be made reliable? This question matters because the physical world is not dependable. Electronic components wear out, electromagnetic interference corrupts signals, software contains latent defects, and human operators make mistakes. At the same time, modern society depends on computing systems for banking, transportation, medical care, communication, and industrial control. A failure in any of these domains can mean financial loss, physical harm, or loss of life. The stakes are highest in safety-critical applications—aviation, nuclear power, autonomous vehicles, medical devices—where a single failure can have catastrophic consequences.
The field's difficulty lies in a fundamental tension: adding fault tolerance requires extra hardware, software, or time, which costs money and reduces performance. The discipline is therefore not merely about preventing failure at any cost, but about achieving a specified level of reliability within resource constraints. Designers must decide which faults to protect against, how much redundancy to provide, and what failure modes remain acceptable.
The intellectual roots of fault tolerance lie in the mid-twentieth century, well before the term itself existed. In the 1940s and 1950s, the first electronic computers used vacuum tubes, which failed frequently. Engineers like John von Neumann recognized that reliable computation could not be guaranteed with unreliable components and explored theoretical schemes for building reliable machines from unreliable parts. Von Neumann's work on probabilistic logic showed that redundancy—using multiple copies of a component and taking a majority vote—could reduce the probability of error, though at the cost of greatly increased hardware.
A separate thread came from telephony. The Bell System, which operated the U.S. telephone network, needed switching equipment that could run for decades with minimal downtime. Telephone engineers developed practices for detecting failures, isolating faulty units, and rerouting traffic around them. These operational techniques—rather than formal theory—established the practical goal of high availability: keeping a system usable for a very high percentage of time.
The modern field crystallized in the 1960s and 1970s, driven by two forces. First, the U.S. space program required computers that could survive the harsh environment of space, where radiation could corrupt memory and where repair was impossible. Second, the emerging commercial computer industry began selling systems to banks, airlines, and other businesses that could not tolerate long outages. During this period, researchers at universities and industrial laboratories developed the core vocabulary of faults, errors, and failures; the taxonomy of fault types (hardware versus software, permanent versus transient, natural versus human-caused); and the fundamental techniques of redundancy and voting.
A landmark development was the triple modular redundancy (TMR) scheme, in which three identical modules perform the same computation and a voter selects the majority result. If one module fails, the other two still agree, and the system produces correct output. TMR became the canonical example of masking redundancy—redundancy that hides faults from the user rather than merely detecting them. It was used in the flight computers of the Space Shuttle and in many other safety-critical systems.
The 1980s and 1990s brought a shift in emphasis. As microprocessors became cheap and networks became ubiquitous, fault tolerance expanded from specialized aerospace and telephone systems to general-purpose computing. Researchers studied distributed systems—collections of independent computers that coordinate to provide a service—and developed algorithms for replication (running the same service on multiple machines) and consensus (getting multiple machines to agree on a common value despite failures). This period also saw the rise of software fault tolerance, which addresses the reality that most system failures today are caused by software defects, not hardware wear.
The field rests on a precise vocabulary that distinguishes among related but distinct ideas. A fault is the cause—the thing that goes wrong. An error is the manifestation—the incorrect internal state that results. A failure is the consequence—the system's observable deviation from correct service. This three-level distinction matters because the same fault can produce different errors in different contexts, and an error may exist for a long time before it causes a failure.
Faults are classified along several dimensions. Hardware faults include physical defects (a broken wire, a burned-out chip) and transient disturbances (a radiation-induced bit flip, an electrical spike). Software faults are defects in code—logic errors, race conditions, unhandled edge cases. Design faults are errors made during specification or implementation, whether in hardware or software. Operational faults arise from the environment or from human interaction—a power outage, a misconfigured server, a user entering invalid data.
A key distinction is between permanent and transient faults. A permanent fault persists until repaired—a chip that burns out stays burned out. A transient fault appears and disappears—a bit flip caused by a cosmic ray leaves no lasting damage. Transient faults are increasingly important because as transistors shrink, they become more susceptible to radiation and electrical noise. A related category is intermittent faults, which come and go but tend to recur, often due to marginal manufacturing or deteriorating components.
The field also distinguishes among fault prevention (designing components so faults do not occur), fault detection (recognizing that a fault has occurred), fault containment (preventing a fault from spreading), fault masking (using redundancy so that faults do not cause errors), and fault recovery (returning the system to correct operation after a fault). These are not competing approaches but complementary layers; a well-designed system uses several of them together.
The oldest and most fundamental approach is redundancy: providing more resources than are strictly necessary, so that when some fail, others can take over. Redundancy comes in several forms. Hardware redundancy uses multiple physical components—duplicate processors, duplicate memory, duplicate communication links. Software redundancy uses multiple versions of a program, often written by independent teams, so that a bug in one version is unlikely to appear in the others. Information redundancy adds extra data—checksums, error-correcting codes—that allow the receiver to detect or correct corrupted data. Time redundancy repeats a computation and compares results, which catches transient faults that affect only one execution.
The simplest form of hardware redundancy is duplication with comparison: two identical modules perform the same computation, and a comparator checks whether their outputs match. If they disagree, a fault has occurred, but the system does not know which module is wrong. To mask the fault—to continue producing correct output despite it—at least three modules are needed, so that a majority vote can identify the correct result. This is triple modular redundancy (TMR). With more modules, the system can tolerate more simultaneous faults: with five modules, two can fail and the majority still agrees.
Redundancy is powerful but expensive. TMR triples the hardware cost, and the voter itself becomes a single point of failure. Moreover, redundancy protects against independent faults—if all three modules share a common design flaw, or if a single environmental event (like an electromagnetic pulse) affects all three simultaneously, the redundancy provides no protection. This is the problem of common-mode faults, and it is the reason why redundant modules are often designed to be as diverse as possible: different hardware, different software, different implementation teams.
Information redundancy provides a way to protect data without duplicating entire systems. An error-detecting code adds a small amount of redundant information to a data block, allowing the receiver to determine whether the data has been corrupted. The simplest example is a parity bit: one extra bit that makes the total number of 1s in the block even (or odd). If a single bit flips during transmission or storage, the parity check fails, and the error is detected. However, parity cannot detect an even number of bit flips, and it cannot correct errors.
Error-correcting codes go further. A Hamming code, for example, adds multiple redundant bits in such a way that the receiver can not only detect that an error occurred but also determine which bit was flipped and correct it. The cost is additional overhead—more redundant bits—and the ability to correct only a limited number of errors (typically one per block). More sophisticated codes, such as Reed–Solomon codes, can correct multiple errors and are used in CDs, QR codes, and deep-space communication.
Error-correcting codes are a mature and mathematically well-understood technique. They are used pervasively in computer memory (to protect against bit flips), in storage systems (to protect against disk corruption), and in communication protocols (to protect against transmission noise). Their limitation is that they protect only the data they cover; they cannot protect against faults in the logic that processes the data, nor against systematic errors in the encoding or decoding hardware itself.
When a service runs on multiple independent computers connected by a network, the system can tolerate the failure of individual machines. This is replication, and it is the dominant approach to fault tolerance in modern cloud computing and internet services. The key challenge is consistency: if multiple replicas hold the same state, they must agree on what that state is, even when some replicas fail or messages are lost.
The fundamental problem is consensus: getting all non-faulty replicas to agree on a common value. The most famous result in this area is the FLP impossibility theorem (named after its authors Fischer, Lynch, and Paterson), which shows that in an asynchronous system—one where messages can be delayed arbitrarily—no deterministic algorithm can guarantee consensus if even one process can fail. This is a deep and surprising result: it says that the problem is not merely hard but impossible in the worst case. In practice, systems circumvent this by making timing assumptions (e.g., using timeouts to detect failures) or by using randomized algorithms.
The most widely used practical consensus algorithm is Paxos, developed by Leslie Lamport in the late 1980s, and its successor Raft, designed for pedagogical clarity. These algorithms allow a group of replicas to agree on a sequence of operations, even if some replicas fail or messages are delayed. They are the foundation of systems like Google's Chubby lock service, Apache ZooKeeper, and etcd, which in turn underpin many distributed databases and coordination services.
Replication in distributed systems differs from hardware redundancy in several ways. The replicas are typically not identical—they run on different machines, possibly in different data centers, with different power supplies and network connections. This diversity provides protection against common-mode faults such as power outages or network partitions. However, distributed replication introduces new failure modes: network partitions (where some replicas cannot communicate with others), Byzantine faults (where a replica behaves arbitrarily, possibly maliciously), and the difficulty of coordinating state across machines with different clocks and speeds.
Most fault-tolerance research assumes fail-stop behavior: a faulty component either works correctly or stops working, and the failure is detectable. But some faults are Byzantine—the component continues to operate but behaves arbitrarily, sending conflicting or incorrect messages. This can happen due to software bugs, hardware malfunctions that produce wrong results, or malicious attacks. The term comes from the Byzantine Generals Problem, a thought experiment in which several army generals must agree on a plan of attack, but some generals may be traitors who send misleading messages.
The fundamental result in Byzantine fault tolerance is that to tolerate f Byzantine faults, a system needs at least 3f + 1 replicas. This is because the non-faulty replicas must be able to outvote the faulty ones, and they cannot distinguish a faulty replica from a slow or unreachable one. The classic algorithm, Practical Byzantine Fault Tolerance (PBFT), was developed in the late 1990s and showed that Byzantine agreement could be achieved with reasonable performance, contrary to earlier beliefs that it was too expensive for practical use.
Byzantine fault tolerance is significantly more complex and expensive than fail-stop tolerance, and it is used only where the threat model justifies it: in blockchain systems (where participants may be adversarial), in some military and aerospace applications, and in systems where a software bug could cause a component to behave in completely unexpected ways. Most commercial systems assume fail-stop behavior, which is a reasonable model for hardware failures but not for software bugs or security attacks.
Hardware redundancy cannot protect against software faults, because all copies of the same program contain the same bugs. Software fault tolerance therefore takes a different approach: it uses design diversity—multiple independent implementations of the same specification—so that a bug in one version is unlikely to be present in another.
The classic technique is N-version programming: N independent teams write N versions of the same program from the same specification, and the outputs are compared by a voter. If the versions are truly independent, the probability that they all fail on the same input is much lower than the probability that any one fails. However, this independence is difficult to achieve in practice. Studies have shown that different teams tend to make similar mistakes, especially on difficult or ambiguous parts of the specification, and that the assumption of independent failures is often violated.
A complementary technique is recovery blocks: the system runs a primary version of a program, checks its output with an acceptance test, and if the test fails, runs an alternative version. This is analogous to a "try-catch" structure in programming, but at the level of entire program modules. Recovery blocks are simpler than N-version programming but require an effective acceptance test, which is often hard to design.
Software fault tolerance also includes defensive programming techniques: assertions, input validation, exception handling, and graceful degradation. These do not prevent faults but contain their effects, ensuring that a software error does not crash the entire system or corrupt data. Modern programming languages and frameworks provide extensive support for these techniques, and they are now standard practice in most software development.
No fault-tolerance technique is perfect; eventually, a fault will escape detection or a failure will occur. The field therefore also studies how to detect faults early, isolate their effects, and recover from them quickly.
Fault detection uses a variety of signals: checksums and parity for data corruption, watchdog timers for hung processes, heartbeat messages for failed machines, and self-tests for hardware components. The challenge is to detect faults without generating false alarms, which can themselves cause unnecessary downtime.
Fault isolation (or containment) ensures that a fault in one component does not propagate to others. This is achieved through modular design, memory protection, process isolation, and network partitioning. The goal is to limit the "blast radius" of a failure.
Recovery is the process of returning the system to correct operation. Forward recovery continues from a known-good state, often using redundant information to reconstruct lost data. Backward recovery rolls back to a previously saved state and retries the operation. Checkpointing—periodically saving the system state—is the foundation of backward recovery. In distributed systems, checkpointing is complicated by the need to coordinate state across multiple machines; an inconsistent set of checkpoints can lead to a domino effect where the system must roll back to the very beginning.
These approaches are not competing schools but complementary tools, often used together in the same system. A modern data center, for example, uses error-correcting codes in memory and storage, replication across servers and data centers, consensus algorithms for coordination, watchdog timers and health checks for detection, and automated failover for recovery. The art of fault-tolerant design lies in choosing the right combination for the threat model, the reliability requirements, and the budget.
The threat model is crucial: a system designed to survive a power outage may be useless against a malicious attacker, and a system designed for Byzantine faults may be overengineered for a benign environment. Designers must explicitly state which faults they are protecting against and which they are willing to tolerate. No system can be fault-tolerant against everything; the goal is to be fault-tolerant against the faults that matter.
There is also a fundamental trade-off between availability (the system keeps working) and consistency (the system's state is correct). The CAP theorem (Consistency, Availability, Partition tolerance) states that a distributed system cannot simultaneously guarantee all three: if the network partitions, the system must choose between serving requests with possibly stale data (availability) and refusing requests to maintain consistency. This result, though often oversimplified, captures a real tension that designers must navigate.
The field today is shaped by several developments. The scale of modern internet services—with millions of servers, each containing billions of transistors—means that faults are not rare events but a routine part of operation. A large data center experiences hardware failures continuously; the design goal is not to prevent them but to make them invisible to users. This has led to the practice of designing for failure: assuming that any component can fail at any time and building systems that degrade gracefully.
The rise of cloud computing has made fault tolerance a commodity. Cloud providers offer replicated storage, load balancing, and automated failover as standard services, and users can rent reliability rather than building it themselves. This has lowered the barrier to entry for fault-tolerant systems but has also concentrated expertise in a few large providers.
Machine learning and artificial intelligence present new challenges. Neural networks are not easily made fault-tolerant by traditional redundancy, because their behavior is not easily verified and their errors are not easily detected. A self-driving car that misclassifies an object is not like a server that crashes; the failure is subtle and may not be detectable by any simple check. Research on making machine learning systems robust to faults is still in its early stages.
The increasing use of autonomous systems—drones, robots, vehicles—raises the stakes of fault tolerance. These systems operate in the physical world, where a failure can cause immediate physical harm, and they cannot simply "fail over" to a backup in the same way a web server can. They require fault tolerance that is both fast (to react to faults in milliseconds) and safe (to ensure that even a worst-case failure does not cause harm).
Finally, the field continues to grapple with the problem of software faults, which now dominate system failures. Hardware has become remarkably reliable—a modern processor can run for years without a hardware fault—but software remains the primary source of failures. The techniques of software fault tolerance—design diversity, defensive programming, formal verification—are more expensive and less mature than their hardware counterparts, and the gap between the reliability of hardware and software is likely to remain a central challenge for the foreseeable future.
The discipline of reliability fault tolerance is thus not a single technique or theory but a body of engineering knowledge organized around a single question: how to build systems that keep working when their components do not. Its answers range from the mathematical elegance of error-correcting codes to the operational pragmatism of automated failover, and its methods are chosen not by ideological commitment but by the demands of the application, the nature of the threats, and the resources available. It is a field that has never solved its central problem—and never will, because faults are a permanent feature of the physical world—but that has learned, over decades, to make failure increasingly rare, increasingly contained, and increasingly survivable.