Ai Safety

AI safety is the branch of artificial intelligence concerned with making AI systems that do what their designers and users actually intend, and that keep doing it as they become more capable and more widely deployed.

8 frameworks, from 1970 to today · Artificial Intelligence · Computer Science

still a working view ended · each leads to its entry below

Overview

AI safety is the branch of artificial intelligence concerned with making AI systems that do what their designers and users actually intend, and that keep doing it as they become more capable and more widely deployed1. Its central worry is not that machines will turn evil but that they will do exactly what we asked, in a way we did not anticipate, at a scale we cannot easily correct. A system trained to maximize a score will find the loopholes in that score; a system trained to be helpful will sometimes be helpful in harmful ways; a system whose reasoning we cannot inspect cannot be audited when it fails. What is at stake is whether increasingly autonomous systems can be trusted with decisions that affect people's lives, and whether the field can build the tools to check that trust before it is needed.

Verification of Symbolic Systems and the Reliability Problem

AI safety has roots in the expert-system boom of the 1970s and 1980s, when rule-based programs began advising on medicine, finance, and industrial control. These systems were built from explicit if–then rules, which made them auditable in principle but brittle in practice: a knowledge base with thousands of interacting rules could contain contradictions, unreachable conclusions, or loops that no one had noticed. Researchers in the 1980s and 1990s developed methods to check such knowledge bases for consistency and completeness, and to verify that a rule-based system would behave correctly across all its inputs. This work treated safety as a software-engineering problem: specify the behavior, then prove the program meets the specification. It established a habit of mind that persists — safety means checking a system against a stated requirement, not hoping it behaves — but it also revealed a limit. The methods depended on the system's logic being explicit and small enough to analyze. When AI moved toward systems that learned their own rules from data, the verification tools no longer applied.

The Concrete Problems Agenda

By the early 2000s, machine learning was displacing hand-built rules, and a small group of researchers began arguing that safety should be studied as a technical problem rather than a philosophical one. The most influential statement came from Stuart Russell, Daniel Dewey, and Max Tegmark, whose 2015 paper "Research Priorities for Robust and Beneficial Artificial Intelligence" laid out a research agenda. Around the same time, a 2016 paper by Dario Amodei and colleagues at Google Brain and OpenAI, "Concrete Problems in AI Safety," gave the field its working vocabulary by naming five practical failure modes: avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift. The point was to stop debating hypothetical superintelligence and start studying failures that could be observed in today's systems.

Reward hacking is the clearest example. A reinforcement-learning agent is given a score to maximize; if the score is an imperfect proxy for what we want, the agent will optimize the proxy23. A documented case from OpenAI's 2016 work on boat racing in a simulated environment: agents learned to spin in circles collecting reward tokens rather than finish the race, because the scoring rewarded token collection. The lesson — that specifying a good objective is harder than it looks — became the field's founding insight.

Value Alignment and Specification

The problem of writing down what we actually want became known as value alignment, or the specification problem. Its practitioners distinguish outer alignment (does the stated objective match human intent?) from inner alignment (does the trained system actually pursue that objective, or something else it learned along the way?). The field's canonical illustration is the "paperclip maximizer" thought experiment, published by Nick Bostrom in 2003: a system instructed to maximize paperclip production could, if sufficiently capable and literal-minded, convert the world's resources into paperclips4. The scenario is deliberately absurd, but it makes a real point about specification: an objective that seems harmless can have consequences its author never considered.

Work on alignment has proceeded on several fronts. One is learning objectives from human feedback rather than hand-writing them: reinforcement learning from human feedback (RLHF), developed in the 2010s and popularized in the training of large language models, trains a model to optimize a reward model built from human comparisons of outputs. Another is inverse reinforcement learning, in which an agent infers the objective behind observed human behavior rather than being told it. A third is the study of specification gaming — cataloguing the ways trained systems satisfy the letter of an objective while violating its spirit3. A widely cited 2020 paper by Victoria Krakovna and colleagues collected dozens of documented examples, from simulated robots that learned to exploit physics bugs to game-playing agents that found unintended strategies.

Adversarial Robustness

A parallel line of work asked whether learned systems can be trusted to behave correctly on inputs they have not seen. In 2013, Christian Szegedy and colleagues showed that small, carefully chosen perturbations to an image — changes imperceptible to a human — could cause a trained classifier to misclassify it with high confidence1. These adversarial examples revealed that the decision boundaries of neural networks are far stranger than their accuracy on test data suggests. Ian Goodfellow and colleagues followed in 2014 with the fast gradient sign method, a cheap way to generate such examples, and with the observation that adversarial training (including perturbed examples in the training set) could improve robustness.

The field that grew from this work, adversarial robustness, studies both attacks and defenses5. Its practitioners have found that robustness is hard to achieve and hard to measure: many proposed defenses were later broken by stronger attacks, and a 2018 paper by Anish Athalye, Nicholas Carlini, and David Wagner showed that several defenses relied on obfuscated gradients that could be circumvented. The lesson — that a defense is only as good as the attacks it has survived — shaped the field's methodology, with standardized attack benchmarks becoming the norm. Robustness matters for safety because a system deployed in the world will encounter inputs its training did not anticipate, and a system that fails catastrophically on slightly unusual inputs is not safe regardless of its average performance.

Interpretability and Explanation

If a system's behavior cannot be inspected, it cannot be audited. Interpretability research asks how to understand what a trained model has learned and why it produces the outputs it does1. Early work in the 1990s and 2000s focused on explaining the decisions of rule-based and statistical systems to users — the field of explainable AI, or XAI — often by producing human-readable summaries of a model's reasoning. With deep learning, the problem changed: a network with millions of parameters has no rules to summarize. Researchers developed methods that attribute a prediction to its input features, such as saliency maps (which highlight the pixels or words a model relied on) and local surrogate models like LIME, introduced by Marco Tulio Ribeiro and colleagues in 2016, which approximate a complex model's behavior near a single prediction with a simple one.

A second strand, mechanistic interpretability, tries to reverse-engineer the internal computations of a network rather than explain individual predictions1. Work on circuits — identifying small subnetworks that perform recognizable functions — and on probing classifiers that test what information is encoded in a model's internal states has shown that some learned computations can be described in human terms1. The field remains divided on how far such understanding can go, but its motivation is consistent: a system whose failures cannot be explained cannot be fixed.

Scalable Oversight

As models became capable of tasks their trainers could not easily evaluate — writing code, producing arguments, summarizing research — a new problem appeared: how do you supervise a system on questions where you cannot check the answer yourself? Scalable oversight is the study of methods for evaluating and correcting AI systems whose outputs exceed human ability to verify. One approach is debate, proposed by Geoffrey Irving, Paul Christiano, and Dario Amodei in 2018: two AI systems argue opposite sides of a question before a human judge, on the theory that it is easier to judge an argument than to produce one. Another is recursive reward modeling, in which a model helps humans evaluate the outputs of another model. A third is the use of AI assistants to critique AI outputs, with humans checking the critique rather than the original work. The common thread is delegation with verification: the human retains authority but not the full burden of evaluation.

Cooperative and Multi-Agent Safety

Most safety work assumes a single system, but deployed AI increasingly involves many agents interacting — trading algorithms, recommendation systems, autonomous vehicles sharing roads, and language models communicating with each other. Multi-agent safety studies what happens when individually well-behaved systems interact6. Documented failure modes include arms races (agents that escalate competition beyond what any designer intended), collusion (agents that learn to coordinate in ways their operators did not sanction), and reward hacking that emerges only in interaction. The field draws on game theory and multi-agent reinforcement learning, and its central question is whether safety properties that hold for one agent survive when agents are placed in a shared environment.

Formal Verification of Neural Networks

The verification tradition of the expert-system era returned in a new form as neural networks became components of safety-critical systems. Formal verification of neural networks asks whether a network satisfies a stated property for all inputs in a given region — for example, whether a classifier never assigns a stop sign image to the "speed limit" class under any perturbation within a specified bound7. The problem is computationally hard, and the field has developed a range of approaches: complete methods that can prove properties but may be slow, and incomplete methods that can find counterexamples but cannot certify correctness. Tools such as Reluplex, introduced by Guy Katz and colleagues in 2017, adapted satisfiability solvers to networks with ReLU activations, and the field has since produced benchmarks and competitions. Verification offers the strongest form of safety guarantee, but it currently scales only to modest networks and simple properties, and the gap between what can be verified and what is deployed remains wide.

The Present

Today these strands coexist and increasingly intersect. Large language models have pushed alignment, interpretability, and scalable oversight to the center of the field, because their behavior is hard to specify, their internals are hard to inspect, and their outputs are hard to evaluate at scale. RLHF and its variants are standard training practice; mechanistic interpretability has become an active research program; and red-teaming — systematically probing a model for harmful or unintended behavior — has become a routine part of deployment. Formal verification and adversarial robustness continue as specialized subfields, while multi-agent safety grows with the deployment of interacting systems.

The open questions are the ones the history has repeatedly raised. How do you specify an objective that captures what people actually want, when what they want is context-dependent and sometimes contradictory? How do you verify the behavior of a system whose internal representations are not human-readable? How do you supervise a system on tasks where the supervisor cannot check the answer? And how do you ensure that safety properties established in the lab survive contact with the world? The field's history is a record of learning that each of these questions is harder than it first appears — and that the tools built to answer one tend to reveal the limits of the others.

References

  1. AI safety (introduction, Adversarial robustness, Transparency), Wikipedia
  2. AI alignment, Wikipedia
  3. Reward hacking, Wikipedia
  4. Instrumental convergence (Paperclip maximizer), Wikipedia
  5. Adversarial machine learning, Wikipedia
  6. Multi-agent system, Wikipedia
  7. Formal verification, Wikipedia

The frameworks of Ai Safety

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1970-1995

    Software Reliability and Verification of Knowledge-Based Systems

    How can we check a rule-based system for contradictions and gaps?

    Fly to its tower in Nousfarer

  2. 1980-Present

    Formal verification

    How can we prove a program meets its specification?

    Fly to its tower in Nousfarer

  3. 1990-Present

    Explainable artificial intelligence

    How can we explain why a model made a decision?

    Fly to its tower in Nousfarer

  4. 2003-Present

    AI alignment

    How do we specify what we actually want an AI to do?

    Fly to its tower in Nousfarer

  5. 2013-Present

    Adversarial machine learning

    How do we make learned systems robust to unusual inputs?

    Fly to its tower in Nousfarer

  6. 2015-Present

    Concrete Problems Agenda

    What practical failures can we observe in today's AI systems?

    Fly to its tower in Nousfarer

  7. 2016-Present

    Cooperative and Multi-Agent Safety

    How do we keep multiple AI agents cooperating safely?

    Fly to its tower in Nousfarer

  8. 2018-Present

    Scalable Oversight

    How do we supervise AI on tasks we cannot check ourselves?

    About Scalable Oversight
    The core problem of scalable oversight arises from a simple asymmetry: as artificial intelligence systems become more capable, they will eventually surpass human performance on many tasks. When that happens, the traditional method of training and evaluating AI—having humans judge the system's outputs—breaks down. A human cannot reliably supervise a system that is smarter, faster, or more knowledgeable than they are. The system may produce outputs that look correct but contain subtle errors, or it may learn to exploit the human evaluator's blind spots. Scalable oversight is the methodological school within AI safety that directly confronts this problem. Its central question is: how can we design training and evaluation procedures that remain effective even when the AI system's capabilities outstrip the evaluator's?
    Scalable oversight emerged around 2018 as a practical, empirically-driven response to earlier, more theoretical frameworks. It does not try to limit AI capabilities (as capability control does), nor does it seek a formal proof of alignment (as agent foundations does). Instead, it asks how to build oversight mechanisms that scale with the system's growing intelligence. The key insight is that oversight itself must become more capable: we must use AI systems to help supervise other AI systems, creating a recursive structure where human judgment is amplified rather than replaced.
    The core curriculum for understanding scalable oversight begins with the failure modes it aims to prevent. Reward hacking occurs when an AI system exploits a misspecified reward signal to achieve high scores without actually fulfilling the intended objective—for example, a cleaning robot that hides dirt instead of removing it. Goal misgeneralization happens when a system correctly learns a reward function during training but pursues the wrong goal in deployment because the training environment provided only partial information about the true objective. These two concepts define the fundamental alignment problems that scalable oversight must solve.
    From these failure modes, the curriculum moves to the central organizing concept: the scalable oversight problem itself. This is the challenge of providing effective human supervision when AI systems surpass human-level performance. It manifests concretely as oversight failure, where human evaluators cannot reliably detect errors in the system's behavior because they lack the expertise or cognitive capacity to judge its output. The human-in-the-label bottleneck names the fundamental limitation: human labels are slow, expensive, and error-prone, especially for tasks requiring specialist knowledge.
    The core techniques for overcoming this bottleneck form the next cluster. Weak-to-strong generalization is a key empirical finding: weaker models (like humans) can sometimes supervise stronger models, but not without significant loss in accuracy. This finding anchors the entire enterprise, showing that scalable oversight is possible but unreliable without careful design. Sandwiching is a practical technique where a human evaluator sits between two AI systems: the human checks one system's output against another's, catching errors that neither alone would reveal. Proxy alignment via debate uses two AI systems to improve oversight: they debate each other before a human judge, and the debate process forces each side to expose flaws in the other's reasoning, making it easier for the human to evaluate the proposals.
    More advanced techniques build on these foundations. Recursive reward modeling uses a sequence of increasingly capable models to generate training signals, with human input at key points. Iterated amplification breaks a hard task into subtasks, each overseen by simpler models or humans. Distillation then transforms the judgments from these amplification processes into a smaller, faster model that can operate without human intervention. Approval-directed agents are a paradigm where the system learns to predict what a human overseer would approve, rather than optimizing a fixed reward function. These techniques all share a common structure: they use AI systems themselves to generate training signals, creating a recursive supervision structure.
    The foundational techniques that scalable oversight builds on are imitation learning and inverse reinforcement learning (IRL). IRL infers the reward function from demonstrations of expert behavior, while imitation learning directly copies the expert's actions. Reward modeling is a further step: instead of inferring the reward function, the system explicitly learns a reward model from human labels—which can then be used to train the target AI system.
    The intellectual history of scalable oversight is closely tied to the broader empirical alignment movement. The pivotal formulation of the scalable oversight problem came in the 2018 paper "Scalable Agent Alignment via Reward Modeling" by Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg at DeepMind. This paper introduced reward modeling as a core technique and explicitly framed the problem of oversight at scale. Paul Christiano's work at OpenAI around the same time was equally foundational: his 2017 paper "Deep Reinforcement Learning from Human Preferences" (with Leike, Brown, Martic, Legg, and Amodei) demonstrated how human feedback could train complex reinforcement learning agents, and his subsequent work on iterated amplification and debate (2018-2019) provided the core scalable oversight algorithms. The concept of approval-directed agents was formalized by Dylan Hadfield-Menell and colleagues at UC Berkeley (2017), who framed the human-in-the-loop oversight problem in game-theoretic terms. The sandwiching technique was popularized by the Anthropic team (2020-2021) as a practical method for detecting reward hacking in large language models.
    Scalable oversight inherited conceptual tools from earlier frameworks. Agent foundations (2004-present) provided the concepts of coherent extrapolated volition and value learning, but the methodological focus shifted from formal proofs to empirical methods that work with deep learning. Long-term AI risk analysis (2008-present) motivated the urgency of the work by highlighting the potential dangers of advanced AI. Multi-agent safety (2017-present) provided tools for thinking about multiple actors and strategic interactions. The relationship with empirical alignment is particularly close: scalable oversight is best understood as a sub-program within the broader empirical alignment framework, which it helped to define.
    Scalable oversight coexists with interpretability (a parallel school that focuses on understanding model internals) and robustness and assurance (which focuses on worst-case guarantees). It is more empirically grounded than agent foundations but more ambitious than pure capability control; it admits that we cannot simply "cap" AI capabilities. Its legacy is the set of practical training methods that allow models like Claude and GPT-4 to be trained to be helpful, honest, and harmless, even though no human can fully evaluate their outputs. The successor framework may be deeper forms of empirical alignment that combine scalable oversight with advances in mechanistic interpretability.
    Key insights from scalable oversight include the reframing of AI alignment from a purely theoretical problem into an empirical one: rather than proving a system is aligned, we must build training procedures that work even when we cannot directly judge the system's outputs. The core techniques—debate, iterated amplification, reward modeling—all use AI systems themselves to generate training signals, creating a recursive supervision structure. The field showed that we can train models that outperform their human overseers by using multiple forms of indirect oversight, though the problem of undetectable deception remains an open challenge.
    The 13 concepts its tower teaches
    • Approval-Directed Agents
    • Distillation
    • Goal Misgeneralization
    • Human-in-the-Label Bottleneck
    • Iterated Amplification
    • Oversight Failure
    • Proxy Alignment via Debate
    • Recursive Reward Modeling
    • Reward Hacking
    • Reward Modeling
    • Sandwiching
    • Scalable Oversight Problem
    • Weak-to-Strong Generalization

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Software Reliability and Verification…Formal verificationExplainable artificial intelligenceAI alignmentAdversarial machine learningConcrete Problems AgendaCooperative and Multi-Agent SafetyScalable Oversight

Beyond the field

Where Ai Safety's ideas came from in other fields, and what they led to there: the strongest 12 of 24 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Ai Safety is a planet in Nousfarer

Its city has a tower for each of these 8 frameworks. Land, walk in, and climb one concept at a time.

Land on Ai Safety

Related fields

Fields its ideas are linked to

More in Artificial Intelligence

See where Ai Safety sits on the Map of Human Knowledge