System Safety

System safety is the branch of engineering that treats accidents as something to be designed out of a system before it is built, rather than something to be discovered and patched after it fails.

15 frameworks, from 1900 to today · Systems Engineering · Engineering

still a working view ended · each leads to its entry below

Overview

System safety is the branch of engineering that treats accidents as something to be designed out of a system before it is built, rather than something to be discovered and patched after it fails1. Its practitioners ask a different question from the quality engineer or the reliability engineer. Reliability asks how often a component will break; system safety asks how a whole arrangement of components, software, people, and procedures can produce harm even when every part is working as intended1. That distinction—between failure and accident—is the thread that runs through the field's history.

The stakes are concrete. A commercial airliner, a nuclear power plant, a chemical refinery, a medical infusion pump, and an autonomous vehicle all contain hazards that cannot be eliminated by making parts more reliable, because the hazard arises from the interaction of parts, or from the way people use the system, or from the software logic itself1. System safety developed the tools to find those hazards early, rank them, and drive them down to an acceptable level. Its history is a story of successive methods, each invented because the previous one missed something.

Fly-Fix-Fly and the Limits of Learning from Crashes

The earliest approach to safety in complex machinery was simply to build it, operate it, and learn from failures. In aviation this was called "fly-fix-fly": a design was flown, crashes were investigated, and the design was modified. The method has real virtues—it is grounded in observed failure—but it is reactive and expensive, and in some domains the cost of a single failure is unacceptable. A nuclear reactor cannot be "flown" to destruction to find its weaknesses.

Through the early twentieth century, safety engineering concentrated on component reliability and on protecting people from obvious mechanical hazards: guards on machinery, pressure relief on boilers, fire suppression. The underlying model was that accidents came from parts breaking or from operator carelessness. That model worked well enough for simple machines but began to fail as systems grew more tightly coupled.

Fault Tree Analysis and the Deductive Turn

A decisive shift came in 1961, when Bell Telephone Laboratories, working on the Minuteman missile program, developed fault tree analysis under the direction of H. A. Watson. A fault tree starts from an undesired top event—say, "warhead detonates inadvertently"—and works backward, decomposing it into the combinations of lower-level failures that could cause it, joined by AND and OR gates2. An OR gate means any one of the inputs suffices; an AND gate means all must occur together. The tree makes the logic of failure explicit and lets engineers see which combinations matter.

Fault tree analysis was a deductive method: it began with the harm and asked what could produce it2. It was soon paired with quantitative reliability data, so that the probability of the top event could be estimated from component failure rates. This made it possible, for the first time, to reason systematically about rare catastrophic events in systems too complex to test to failure.

Failure Modes and Effects Analysis and the Inductive View

Where fault trees reason downward from a bad outcome, failure modes and effects analysis (FMEA) reasons upward from each component3. Developed in the U.S. military in the late 1940s and formalized in the 1950s—its procedure was codified in military specification MIL-P-1629 in 1949—FMEA asks, for each part, "how can this part fail, and what happens to the system if it does?" Each failure mode is rated for severity, likelihood, and detectability, and the ratings are multiplied into a risk priority number that ranks what to fix first.

FMEA and fault tree analysis became complementary: FMEA is inductive and exhaustive over components, fault tree analysis is deductive and focused on a specific hazard3. Together they formed the analytical core of system safety for decades. A related method, hazard and operability study (HAZOP), emerged in the British chemical industry in the 1960s, developed at ICI and published in a 1974 guide by Trevor Kletz. HAZOP applies guide words—"no," "more," "less," "reverse," "as well as"—to each part of a process design to provoke questions about deviations from intent. It proved especially powerful for continuous processes like refineries, where the hazard is not a part breaking but a flow, pressure, or temperature going wrong.

The Hazard Control Hierarchy and the Emergence of a Discipline

As these methods spread, a body of doctrine grew around them. The U.S. Department of Defense issued MIL-STD-882 in 1969, the first formal system safety standard, requiring contractors to perform hazard analysis across a program's life. The standard introduced a way of thinking that became central: hazards should be controlled in a preferred order. First, eliminate the hazard by design—remove the flammable material, separate the fuel from the ignition source4. If that is impossible, add safety devices and guards4. If that is still insufficient, provide warning and training4. Personal protective equipment is the last resort4. This "hierarchy of controls" reflects a hard-won lesson: procedural and protective measures fail more often than design changes, so design changes are preferred4.

MIL-STD-882 went through several revisions, and its descendants remain in use. The discipline it helped create—system safety engineering—was institutionalized in professional societies and in university teaching, and it established the idea that safety is a property to be engineered into a system from the start, tracked through a hazard tracking system, and verified before operation.

Human Factors: The Operator as Part of the System

Early system safety treated the human operator mainly as a source of error to be controlled. That changed as researchers studied how people actually interact with machines. Human factors engineering, growing out of World War II aviation psychology, showed that many "operator errors" were really design errors: controls that invited mistakes, displays that hid critical information, workloads that exceeded attention. The field's insight was that the human is a component of the system, and like any component, the human's capabilities and limits must be designed for.

This reframing had practical consequences. It led to design principles for alarms, for control layouts, and for the allocation of functions between people and automation. It also complicated the older assumption that safety could be achieved by making the human more careful. If the human is part of the system, then the system's design—not just the human's vigilance—is the object of safety work.

Software Changes the Problem

From the 1970s onward, software entered safety-critical systems—flight controls, reactor protection, medical devices—and broke the assumptions of the existing methods. Fault tree analysis and FMEA were built for hardware, where failure modes are physical and failure rates can be measured. Software does not wear out, does not have a failure rate in the same sense, and can fail in ways no component-level analysis predicts: a logic error that only manifests in a particular sequence of inputs, or an interaction between two correct modules that produces a wrong output.

Software safety emerged as a distinct concern, with its own techniques: formal specification, rigorous code review, defensive programming, and analysis of software requirements for hazardous behavior. The field learned that software could introduce hazards that no hardware analysis would catch, and that the safety case for a software-intensive system had to address the software's logic, not just its reliability.

Probabilistic Risk Assessment and the Normal Accident Debate

The 1970s brought a quantitative ambition. Probabilistic risk assessment (PRA) sought to estimate the probability and consequence of severe accidents by combining fault trees, event trees, and component reliability data into a single model. Its landmark application was the Reactor Safety Study (WASH-1400), published by the U.S. Nuclear Regulatory Commission in 1975 under Norman Rasmussen. The study attempted to estimate the risk of a severe reactor accident and became the template for risk-informed regulation.

PRA provoked a lasting argument about whether complex systems can be made safe by analysis at all. In 1984, Charles Perrow published Normal Accidents, arguing that in systems with high interactive complexity and tight coupling—nuclear plants, chemical plants, air traffic control—accidents are not anomalies but normal, inevitable products of the system's structure5. Perrow's claim was that some systems are so complex that no designer can foresee all the interactions, and so tightly coupled that failures propagate too fast for operators to intervene5. The argument challenged the confidence of quantitative risk assessment and reframed safety as a property of system architecture, not just of component reliability.

Organizational Accidents and the Swiss Cheese Model

A parallel line of thinking located accidents in organizations rather than in components or operators. In 1990, James Reason proposed the "Swiss cheese model," in which a system's defenses are like slices of cheese, each with holes representing weaknesses. An accident occurs when the holes in successive layers line up, allowing a hazard to pass through every defense. The model's value was that it explained how multiple small failures—none sufficient alone—combine into disaster, and it directed attention to latent conditions: organizational decisions, like understaffing or deferred maintenance, that create holes long before an accident.

Reason's work, along with studies of disasters such as Chernobyl and the Challenger loss, established the idea of the organizational accident. Safety was no longer only a matter of hardware and procedures; it was a matter of management, culture, and the conditions under which people work. This thinking fed directly into safety management systems—formal, audited frameworks for managing safety as an organizational process, adopted in aviation, rail, and process industries—and into the "safety culture" concept that became widespread after the 1986 Chernobyl disaster.

Functional Safety and the Safety Lifecycle

As software and electronics spread into safety functions, a new regulatory approach emerged: functional safety, which governs the safety-related functions a system performs rather than the specific technology implementing them. The central idea is the safety lifecycle—a structured process running from hazard analysis through design, verification, operation, and decommissioning—and the assignment of a safety integrity level (SIL) to each safety function, reflecting how much risk reduction it must provide.

The key standard is IEC 61508, published in its first edition in 1998, which established the safety lifecycle and SIL framework for electrical, electronic, and programmable electronic safety-related systems. Sector-specific standards followed, such as IEC 61511 for process industries and ISO 26262 for automotive67. These standards made system safety auditable and certifiable, and they embedded the discipline's core methods—hazard analysis, risk ranking, verification—into legally binding engineering practice.

New Models for a Software-Intensive World

By the 2000s, a growing critique held that the classical methods, built on chains of component failure, did not fit software-intensive and socio-technical systems. Nancy Leveson of MIT argued that accidents increasingly arise not from component failures but from unsafe control: a controller—human or software—issues a command that is unsafe given the system's state, or fails to issue a command that was needed. Her system-theoretic accident model and processes (STAMP) treats safety as a control problem, and its analysis method, system-theoretic process analysis (STPA), identifies unsafe control actions and the conditions that could produce them.

STPA's appeal is that it does not require a failure to exist; it asks what unsafe interactions are possible given the system's design and its controllers. It has been adopted in aviation, automotive, and defense, and it represents the field's latest attempt to keep pace with systems whose complexity outruns component-level analysis.

The Present

System safety today is a layered practice. Classical techniques—fault tree analysis, FMEA, HAZOP—remain in daily use and are embedded in standards. Probabilistic risk assessment underpins regulation in nuclear and other high-hazard industries. Functional safety standards govern the certification of safety-critical electronics and software. Organizational approaches—safety management systems, safety culture, the Swiss cheese model—shape how companies govern safety. And newer control-theoretic methods like STPA address the software-intensive, tightly coupled systems that the older tools handle poorly.

The open questions are the ones the history keeps returning to: how to analyze systems whose behavior emerges from interactions no one designed, how to certify software whose failure modes are not physical, and how to reconcile quantitative risk estimates with the argument that some systems are too complex to fully foresee. The field's enduring conviction—that safety is a design property, established early and verified continuously—remains its organizing principle.

References

  1. System safety, Wikipedia
  2. Fault tree analysis, Wikipedia
  3. Failure mode and effects analysis, Wikipedia
  4. Hierarchy of hazard controls, Wikipedia
  5. Normal Accidents, Wikipedia
  6. Software safety, Wikipedia
  7. IEC 61511 (introduction, History), Wikipedia

The frameworks of System Safety

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1900-1960

    Fly-Fix-Fly

    How do we learn safety from crashes?

    Fly to its tower in Nousfarer

  2. 1945-Present

    Ergonomics

    How do people interact with machines?

    Fly to its tower in Nousfarer

  3. 1949-Present
  4. 1960-Present

    Hazard and Operability Study

    What deviations from intent can occur?

    Fly to its tower in Nousfarer

  5. 1961-Present

    Fault Tree Analysis

    What combinations cause a bad outcome?

    Fly to its tower in Nousfarer

  6. 1962-Present
  7. 1969-Present

    Hierarchy of hazard controls

    In what order should hazards be controlled?

    Fly to its tower in Nousfarer

  8. 1970-Present

    Software Safety

    How can software cause accidents?

    Fly to its tower in Nousfarer

  9. 1975-Present

    Probabilistic Risk Assessment

    How likely and severe is a catastrophic accident?

    Fly to its tower in Nousfarer

  10. 1984-Present

    System accident

    Are some accidents inevitable in complex systems?

    Fly to its tower in Nousfarer

  11. 1990-Present

    Organizational Accident Model

    How do organizations produce accidents?

    Fly to its tower in Nousfarer

  12. 1990-Present

    Safety management system

    How should safety be managed organizationally?

    Fly to its tower in Nousfarer

  13. 1998-Present

    Functional Safety

    How do we assure safety functions work?

    Fly to its tower in Nousfarer

  14. 2000-Present
  15. 2011-Present

    System-Theoretic Process Analysis

    How do we identify unsafe control actions?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Fly-Fix-FlyErgonomicsFailure mode and effects analysisHazard and Operability StudyFault Tree AnalysisSystem Safety Engineering and Hazard …Hierarchy of hazard controlsSoftware SafetyProbabilistic Risk AssessmentSystem accidentOrganizational Accident ModelSafety management systemFunctional SafetySystem-Theoretic Accident Model and P…System-Theoretic Process Analysis

Beyond the field

Where System Safety's ideas came from in other fields, and what they led to there: the strongest 12 of 65 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

System Safety is a planet in Nousfarer

Its city has a tower for each of these 15 frameworks. Land, walk in, and climb one concept at a time.

Land on System Safety

Related fields

Fields its ideas are linked to

More in Systems Engineering

See where System Safety sits on the Map of Human Knowledge