Loading field map
The subfield of AI safety emerged from the broader discipline of Artificial Intelligence as researchers began to systematically consider the potential risks posed by advanced AI systems. Its central questions revolve around how to ensure that increasingly capable AI systems are aligned with human values and intentions, remain under meaningful human control, and do not cause catastrophic harm, especially as they approach or surpass human-level capabilities across general domains. The history of the field is marked by a transition from speculative, long-term concerns to concrete, near-term technical research programmes, accompanied by the development of distinct, durable paradigms and methodological schools.
Early foundational work in the 2000s and early 2010s was largely philosophical and theoretical, establishing the core problem space. This Agent Foundations research programme sought to formally analyze the properties of intelligent agents, their goals, and their potential behaviors over long time horizons. It asked abstract questions about value specification, goal stability, and the incentives of superintelligent systems. This tradition, often associated with the work of thinkers like Eliezer Yudkowsky and academic philosophers like Nick Bostrom, framed the problem as one of Long-Term Trajectory Analysis, focusing on existential risk scenarios and the challenges of controlling systems vastly more intelligent than humanity.
By the mid-2010s, as machine learning—particularly deep learning—began achieving dramatic empirical successes, the field underwent a pivotal methodological shift. The Empirical Alignment paradigm arose, arguing that safety problems must be tackled empirically on existing AI systems to develop scalable techniques that could generalize to more powerful models. This created a new, engineering-focused research culture centered on contemporary neural networks. This shift did not replace long-term concerns but operationalized them into near-term experiments. Within this empirical turn, several major peer frameworks crystallized.
Capability Control (or "boxing") emerged as a framework focused on external constraints, such as containment, monitoring, and tripwires, to limit an AI system's ability to cause harm even if it is misaligned. In contrast, the Robustness and Assurance framework concentrated on making AI systems themselves more reliable, verifiable, and secure against failures, adversarial attacks, and distributional shifts. A third major framework, Multi-Agent Safety, grew to address the distinct challenges arising from many AI systems interacting, including competition, cooperation, and emergent societal-scale effects.
Concurrently, specific methodological schools became central to the field's technical agenda. Interpretability (or "Explainable AI" in a safety context) developed as a school dedicated to reverse-engineering the internal representations and decision-making processes of complex models to detect misalignment or deception. Scalable Oversight formed as a school tackling the problem of supervising AI systems that may outperform human supervisors on complex tasks, developing techniques like debate, recursive reward modeling, and assisted oversight.
The late 2010s and early 2020s saw these paradigms and schools tested and refined on increasingly capable large language models and other foundation models. The empirical alignment paradigm became dominant, with interpretability and scalable oversight serving as its primary methodological engines. Research expanded into areas like Representational Alignment, which probes and steers internal model concepts, and Out-of-Distribution Robustness. Governance, policy, and standardization efforts gained prominence but remained distinct from the core technical paradigms. The current landscape is characterized by intense research within the empirical alignment paradigm, with ongoing tension between near-term robustness work and long-term agent-focused analysis, and increasing integration of insights from multi-agent systems and cybersecurity into the core safety agenda.
###