- Atlas
- Engineering
- System Safety
System Safety
System safety is the branch of engineering that treats accidents as something to be designed out of a system before it is built, rather than something to be discovered and patched after it fails.
15 frameworks, from 1900 to today · Systems Engineering · Engineering
- Fly-Fix-Fly 1900-1960
- Ergonomics 1945-Present
- Failure mode and effects analysis 1949-Present
- Hazard and Operability Study 1960-Present
- Fault Tree Analysis 1961-Present
- System Safety Engineering and Hazard Control Hierarchy 1962-Present
- Hierarchy of hazard controls 1969-Present
- Software Safety 1970-Present
- Probabilistic Risk Assessment 1975-Present
- System accident 1984-Present
- Organizational Accident Model 1990-Present
- Safety management system 1990-Present
- Functional Safety 1998-Present
- System-Theoretic Accident Model and Processes 2000-Present
- System-Theoretic Process Analysis 2011-Present
Overview
System safety is the branch of engineering that treats accidents as something to be designed out of a system before it is built, rather than something to be discovered and patched after it fails1. Its practitioners ask a different question from the quality engineer or the reliability engineer. Reliability asks how often a component will break; system safety asks how a whole arrangement of components, software, people, and procedures can produce harm even when every part is working as intended1. That distinction—between failure and accident—is the thread that runs through the field's history.
The stakes are concrete. A commercial airliner, a nuclear power plant, a chemical refinery, a medical infusion pump, and an autonomous vehicle all contain hazards that cannot be eliminated by making parts more reliable, because the hazard arises from the interaction of parts, or from the way people use the system, or from the software logic itself1. System safety developed the tools to find those hazards early, rank them, and drive them down to an acceptable level. Its history is a story of successive methods, each invented because the previous one missed something.
Fly-Fix-Fly and the Limits of Learning from Crashes
The earliest approach to safety in complex machinery was simply to build it, operate it, and learn from failures. In aviation this was called "fly-fix-fly": a design was flown, crashes were investigated, and the design was modified. The method has real virtues—it is grounded in observed failure—but it is reactive and expensive, and in some domains the cost of a single failure is unacceptable. A nuclear reactor cannot be "flown" to destruction to find its weaknesses.
Through the early twentieth century, safety engineering concentrated on component reliability and on protecting people from obvious mechanical hazards: guards on machinery, pressure relief on boilers, fire suppression. The underlying model was that accidents came from parts breaking or from operator carelessness. That model worked well enough for simple machines but began to fail as systems grew more tightly coupled.
Fault Tree Analysis and the Deductive Turn
A decisive shift came in 1961, when Bell Telephone Laboratories, working on the Minuteman missile program, developed fault tree analysis under the direction of H. A. Watson. A fault tree starts from an undesired top event—say, "warhead detonates inadvertently"—and works backward, decomposing it into the combinations of lower-level failures that could cause it, joined by AND and OR gates2. An OR gate means any one of the inputs suffices; an AND gate means all must occur together. The tree makes the logic of failure explicit and lets engineers see which combinations matter.
Fault tree analysis was a deductive method: it began with the harm and asked what could produce it2. It was soon paired with quantitative reliability data, so that the probability of the top event could be estimated from component failure rates. This made it possible, for the first time, to reason systematically about rare catastrophic events in systems too complex to test to failure.
Failure Modes and Effects Analysis and the Inductive View
Where fault trees reason downward from a bad outcome, failure modes and effects analysis (FMEA) reasons upward from each component3. Developed in the U.S. military in the late 1940s and formalized in the 1950s—its procedure was codified in military specification MIL-P-1629 in 1949—FMEA asks, for each part, "how can this part fail, and what happens to the system if it does?" Each failure mode is rated for severity, likelihood, and detectability, and the ratings are multiplied into a risk priority number that ranks what to fix first.
FMEA and fault tree analysis became complementary: FMEA is inductive and exhaustive over components, fault tree analysis is deductive and focused on a specific hazard3. Together they formed the analytical core of system safety for decades. A related method, hazard and operability study (HAZOP), emerged in the British chemical industry in the 1960s, developed at ICI and published in a 1974 guide by Trevor Kletz. HAZOP applies guide words—"no," "more," "less," "reverse," "as well as"—to each part of a process design to provoke questions about deviations from intent. It proved especially powerful for continuous processes like refineries, where the hazard is not a part breaking but a flow, pressure, or temperature going wrong.
The Hazard Control Hierarchy and the Emergence of a Discipline
As these methods spread, a body of doctrine grew around them. The U.S. Department of Defense issued MIL-STD-882 in 1969, the first formal system safety standard, requiring contractors to perform hazard analysis across a program's life. The standard introduced a way of thinking that became central: hazards should be controlled in a preferred order. First, eliminate the hazard by design—remove the flammable material, separate the fuel from the ignition source4. If that is impossible, add safety devices and guards4. If that is still insufficient, provide warning and training4. Personal protective equipment is the last resort4. This "hierarchy of controls" reflects a hard-won lesson: procedural and protective measures fail more often than design changes, so design changes are preferred4.
MIL-STD-882 went through several revisions, and its descendants remain in use. The discipline it helped create—system safety engineering—was institutionalized in professional societies and in university teaching, and it established the idea that safety is a property to be engineered into a system from the start, tracked through a hazard tracking system, and verified before operation.
Human Factors: The Operator as Part of the System
Early system safety treated the human operator mainly as a source of error to be controlled. That changed as researchers studied how people actually interact with machines. Human factors engineering, growing out of World War II aviation psychology, showed that many "operator errors" were really design errors: controls that invited mistakes, displays that hid critical information, workloads that exceeded attention. The field's insight was that the human is a component of the system, and like any component, the human's capabilities and limits must be designed for.
This reframing had practical consequences. It led to design principles for alarms, for control layouts, and for the allocation of functions between people and automation. It also complicated the older assumption that safety could be achieved by making the human more careful. If the human is part of the system, then the system's design—not just the human's vigilance—is the object of safety work.
Software Changes the Problem
From the 1970s onward, software entered safety-critical systems—flight controls, reactor protection, medical devices—and broke the assumptions of the existing methods. Fault tree analysis and FMEA were built for hardware, where failure modes are physical and failure rates can be measured. Software does not wear out, does not have a failure rate in the same sense, and can fail in ways no component-level analysis predicts: a logic error that only manifests in a particular sequence of inputs, or an interaction between two correct modules that produces a wrong output.
Software safety emerged as a distinct concern, with its own techniques: formal specification, rigorous code review, defensive programming, and analysis of software requirements for hazardous behavior. The field learned that software could introduce hazards that no hardware analysis would catch, and that the safety case for a software-intensive system had to address the software's logic, not just its reliability.
Probabilistic Risk Assessment and the Normal Accident Debate
The 1970s brought a quantitative ambition. Probabilistic risk assessment (PRA) sought to estimate the probability and consequence of severe accidents by combining fault trees, event trees, and component reliability data into a single model. Its landmark application was the Reactor Safety Study (WASH-1400), published by the U.S. Nuclear Regulatory Commission in 1975 under Norman Rasmussen. The study attempted to estimate the risk of a severe reactor accident and became the template for risk-informed regulation.
PRA provoked a lasting argument about whether complex systems can be made safe by analysis at all. In 1984, Charles Perrow published Normal Accidents, arguing that in systems with high interactive complexity and tight coupling—nuclear plants, chemical plants, air traffic control—accidents are not anomalies but normal, inevitable products of the system's structure5. Perrow's claim was that some systems are so complex that no designer can foresee all the interactions, and so tightly coupled that failures propagate too fast for operators to intervene5. The argument challenged the confidence of quantitative risk assessment and reframed safety as a property of system architecture, not just of component reliability.
Organizational Accidents and the Swiss Cheese Model
A parallel line of thinking located accidents in organizations rather than in components or operators. In 1990, James Reason proposed the "Swiss cheese model," in which a system's defenses are like slices of cheese, each with holes representing weaknesses. An accident occurs when the holes in successive layers line up, allowing a hazard to pass through every defense. The model's value was that it explained how multiple small failures—none sufficient alone—combine into disaster, and it directed attention to latent conditions: organizational decisions, like understaffing or deferred maintenance, that create holes long before an accident.
Reason's work, along with studies of disasters such as Chernobyl and the Challenger loss, established the idea of the organizational accident. Safety was no longer only a matter of hardware and procedures; it was a matter of management, culture, and the conditions under which people work. This thinking fed directly into safety management systems—formal, audited frameworks for managing safety as an organizational process, adopted in aviation, rail, and process industries—and into the "safety culture" concept that became widespread after the 1986 Chernobyl disaster.
Functional Safety and the Safety Lifecycle
As software and electronics spread into safety functions, a new regulatory approach emerged: functional safety, which governs the safety-related functions a system performs rather than the specific technology implementing them. The central idea is the safety lifecycle—a structured process running from hazard analysis through design, verification, operation, and decommissioning—and the assignment of a safety integrity level (SIL) to each safety function, reflecting how much risk reduction it must provide.
The key standard is IEC 61508, published in its first edition in 1998, which established the safety lifecycle and SIL framework for electrical, electronic, and programmable electronic safety-related systems. Sector-specific standards followed, such as IEC 61511 for process industries and ISO 26262 for automotive67. These standards made system safety auditable and certifiable, and they embedded the discipline's core methods—hazard analysis, risk ranking, verification—into legally binding engineering practice.
New Models for a Software-Intensive World
By the 2000s, a growing critique held that the classical methods, built on chains of component failure, did not fit software-intensive and socio-technical systems. Nancy Leveson of MIT argued that accidents increasingly arise not from component failures but from unsafe control: a controller—human or software—issues a command that is unsafe given the system's state, or fails to issue a command that was needed. Her system-theoretic accident model and processes (STAMP) treats safety as a control problem, and its analysis method, system-theoretic process analysis (STPA), identifies unsafe control actions and the conditions that could produce them.
STPA's appeal is that it does not require a failure to exist; it asks what unsafe interactions are possible given the system's design and its controllers. It has been adopted in aviation, automotive, and defense, and it represents the field's latest attempt to keep pace with systems whose complexity outruns component-level analysis.
The Present
System safety today is a layered practice. Classical techniques—fault tree analysis, FMEA, HAZOP—remain in daily use and are embedded in standards. Probabilistic risk assessment underpins regulation in nuclear and other high-hazard industries. Functional safety standards govern the certification of safety-critical electronics and software. Organizational approaches—safety management systems, safety culture, the Swiss cheese model—shape how companies govern safety. And newer control-theoretic methods like STPA address the software-intensive, tightly coupled systems that the older tools handle poorly.
The open questions are the ones the history keeps returning to: how to analyze systems whose behavior emerges from interactions no one designed, how to certify software whose failure modes are not physical, and how to reconcile quantitative risk estimates with the argument that some systems are too complex to fully foresee. The field's enduring conviction—that safety is a design property, established early and verified continuously—remains its organizing principle.
References
- System safety, Wikipedia
- Fault tree analysis, Wikipedia
- Failure mode and effects analysis, Wikipedia
- Hierarchy of hazard controls, Wikipedia
- Normal Accidents, Wikipedia
- Software safety, Wikipedia
- IEC 61511 (introduction, History), Wikipedia
The frameworks of System Safety
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1900-1960
- 1945-Present
- 1949-Present
Failure mode and effects analysis
How can each part fail and what happens?
- 1960-Present
- 1961-Present
- 1962-Present
System Safety Engineering and Hazard Control Hierarchy
How do we engineer safety into a system?
- 1969-Present
Hierarchy of hazard controls
In what order should hazards be controlled?
- 1970-Present
- 1975-Present
Probabilistic Risk Assessment
How likely and severe is a catastrophic accident?
- 1984-Present
- 1990-Present
- 1990-Present
- 1998-Present
- 2000-Present
System-Theoretic Accident Model and Processes
How do control failures lead to accidents?
- 2011-Present
System-Theoretic Process Analysis
How do we identify unsafe control actions?
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
System Safety Engineering and Hazard Control Hierarchy reacted against Fly-Fix-Fly
System safety engineering emerged as a deliberate alternative to fly-fix-fly, arguing that hazards must be identified and controlled by design before operation rather than discovered through crashes.
Carried by MIL-STD-882 (1969).
Probabilistic Risk Assessment influenced by Fault Tree Analysis
Probabilistic risk assessment combined fault trees with event trees and reliability data to quantify severe accident risk, as in the Reactor Safety Study (WASH-1400).
Carried by Norman Rasmussen; Reactor Safety Study (1975).
System accident reacted against Probabilistic Risk Assessment
Charles Perrow's normal accident theory attacked the confidence of probabilistic risk assessment, arguing that in tightly coupled, interactively complex systems accidents are inevitable and cannot be fully foreseen by analysis.
Carried by Charles Perrow; Normal Accidents (1984).
Organizational Accident Model influenced by System accident
Organizational accident models built on Perrow's insight that accidents arise from system structure and interaction, extending the analysis to organizational and managerial factors.
Carried by James Reason.
System-Theoretic Accident Model and Processes influenced by System accident
STAMP took up Perrow's emphasis on interaction and coupling, recasting accidents as control problems rather than chains of component failures.
Carried by Nancy Leveson.
System-Theoretic Process Analysis grew out of System-Theoretic Accident Model and Processes
STPA is the hazard analysis method derived directly from STAMP's control-theoretic accident model, identifying unsafe control actions and flawed control loops.
Carried by Nancy Leveson.
Fault Tree Analysis rival of Failure mode and effects analysis
FMEA and fault tree analysis developed as complementary but competing analytical approaches in the late 1950s and early 1960s: FMEA inductive and component-focused, FTA deductive and hazard-focused.
Carried by H. A. Watson.
Software Safety grew out of System Safety Engineering and Hazard Control Hierarchy
Software safety extended system safety engineering's hazard analysis and control principles to software, treating software logic as a source of hazards to be designed out.
Safety management system influenced by System Safety Engineering and Hazard Control Hierarchy
Safety management systems institutionalized system safety engineering's requirement for hazard tracking, verification, and lifecycle safety assurance within organizations.
System-Theoretic Accident Model and Processes influenced by Software Safety
STAMP was motivated in part by the inadequacy of hardware-oriented methods for software-intensive systems, building on software safety's recognition that software hazards arise from flawed logic and interaction.
Carried by Nancy Leveson.
System-Theoretic Accident Model and Processes influenced by Organizational Accident Model
STAMP incorporated organizational and managerial factors from organizational accident models into its control-theoretic hierarchy of safety constraints.
Carried by Nancy Leveson.
Beyond the field
Where System Safety's ideas came from in other fields, and what they led to there: the strongest 12 of 65 links.
Ergonomics led to Resilience engineering in Human Factors
Resilience engineering grew out of the human-factors and system-safety tradition, especially the cognitive systems engineering work of Hollnagel and Woods on joint cognitive systems.
Carried by Erik Hollnagel; David Woods.
Ergonomics grew out of Human-Machine Systems Engineering in Human Factors
Human-machine systems engineering, developed in WWII military research on operators of radar and aircraft, was the direct parent of postwar ergonomics. The discipline formalized the operator-machine interface that ergonomics then generalized to civilian work.
System Safety Engineering and Hazard Control Hierarchy led to Standards-based safety certification in Systems Verification
System safety engineering and its hazard control hierarchy are the intellectual foundation of safety certification standards such as those for airborne and railway systems. The standards formalize hazard analysis and mitigation ordering into certifiable requirements.
System Safety Engineering and Hazard Control Hierarchy influenced Safety management system in Workplace Health And Safety
System safety engineering, especially the hazard control hierarchy and MIL-STD-882 style requirements, provided the technical core that safety management systems formalized into organizational policy, risk assessment and assurance processes.
Carried by MIL-STD-882; Jerome Lederer.
Software Safety led to Standards-based safety certification in Systems Verification
Software safety grew out of system safety engineering applied to software, and standards-based certification of software-intensive systems is its codification. Safety standards for software define the certification criteria that this framework enforces.
Software Safety influenced by Reliability Engineering in Reliability Fault Tolerance
Software safety grew out of reliability engineering's concepts of failure modes, redundancy and quantitative reliability, applied to software. Software reliability engineering and safety-critical software practice extended reliability methods to programs.
Carried by Reliability Engineering.
Software Safety led to Mixed criticality in Embedded Systems
Mixed criticality extends software safety practice: it asks how to certify high-criticality software while co-hosting lower-criticality software on shared hardware, a question that grows out of safety-assurance methods. The framework is a refinement of software safety for shared-resource platforms.
System accident influenced Systems Approach to Patient Safety in Health Services Research
The system accident concept from system safety engineering, especially Perrow's normal accident theory of 1984, supplied the idea that accidents arise from interactions across a system rather than from individual failure. The systems approach to patient safety imported this framing into healthcare.
Carried by Charles Perrow; Normal Accidents (1984).
Fly-Fix-Fly provoked Reliability-Centered Maintenance in Reliability Engineering
Reliability-centred maintenance arose from dissatisfaction with fly-fix-fly and fixed-interval overhaul practice in aviation; it replaced scheduled restoration with condition-based, consequence-driven maintenance decisions.
Carried by Nowlan; Heap.
Fly-Fix-Fly replaced by Fail-safe design in Aerospace Structures
Fail-safe design replaced fly-fix-fly by building in redundant load paths and inspectable structure so that a single failure would not be catastrophic. It was a deliberate move away from correcting failures only after crashes.
Fly-Fix-Fly replaced by Bathtub Curve and Failure-Rate Regimes in Reliability Engineering
The fly-fix-fly approach was the empirical trial-and-error practice whose limitations prompted the formal failure-rate regime analysis and Weibull plotting that reliability engineering replaced it with.
Carried by Waloddi Weibull.
Fly-Fix-Fly influenced Test-and-fix verification in Systems Verification
Fly-Fix-Fly, the practice of testing a system in operation and correcting failures as they appear, is the engineering model that test-and-fix verification formalised for software. The cycle of run, observe failure, repair, and rerun carried directly into early program testing.

System Safety is a planet in Nousfarer
Its city has a tower for each of these 15 frameworks. Land, walk in, and climb one concept at a time.
Land on System SafetyRelated fields
Fields its ideas are linked to
- Reliability Engineering 11 links
- Human Factors 6 links
- Aerospace Structures 5 links
- Workplace Health And Safety 5 links
- Systems Verification 5 links
- Patient Safety 4 links
- Process Design 3 links
- Ai Safety 2 links
- Chemical Engineering 2 links
- Embedded Systems 2 links
- Disaster History 2 links
- Occupational Health 2 links