Systems Security

Systems security is the branch of cybersecurity concerned with the machine itself: the operating system, the hardware beneath it, the configuration it runs in, and the people who administer and use it.

11 frameworks, from 1972 to today · Cybersecurity · Computer Science

still a working view ended · each leads to its entry below

Overview

Systems security is the branch of cybersecurity concerned with the machine itself: the operating system, the hardware beneath it, the configuration it runs in, and the people who administer and use it. Where network security guards the links between computers and software security guards individual programs, systems security asks whether the whole assembled platform can be trusted to enforce a policy—who may read what, which code may run, and whether anyone can tell when something has gone wrong. Its history is a long argument about where to put the boundary of trust, and what to do when that boundary turns out to be in the wrong place.

The reference monitor and the trusted computing base

The field's founding idea came from an operational problem. In the 1960s, time-sharing systems let many users run jobs on one machine, and the military wanted to put classified work on shared computers without leaking secrets between cleared and uncleared users12. The U.S. Air Force sponsored a study, published in 1972 as the Anderson Report, that framed the question sharply: a system is secure only if every access by a subject to an object is mediated by a small, tamper-proof, always-invoked piece of hardware and software. This was the reference monitor—an abstract enforcement point—and the collection of components that must be trusted to implement it became the trusted computing base (TCB)34. The TCB concept's power was diagnostic: it told designers that security depended on the size and correctness of a specific, identifiable part of the system, not on the whole thing being well built.

The idea was made concrete in the early 1970s at MIT's Project MAC, where Jerome Saltzer and Michael Schroeder, working on the Multics system, distilled a set of design principles—economy of mechanism, fail-safe defaults, complete mediation, open design, separation of privilege, least privilege, least common mechanism, psychological acceptability—that remain the field's catechism. Multics itself, a joint MIT–Bell Labs–General Electric effort, was the first operating system designed from the ground up around protection rings and per-segment access control, and its ring architecture directly influenced the Intel 80286 and later x86 privilege levels.

Multilevel security and mandatory access control

The military's need produced a formal theory. In 1973, David Bell and Leonard LaPadula at MITRE published a model of multilevel security in which every subject and object carries a classification, and access is governed by two rules: no read up (a secret-cleared user cannot read top-secret data) and no write down (a top-secret process cannot write to a secret file). This mandatory access control (MAC) contrasted with discretionary access control (DAC), in which the owner of a file decides who may access it—the model inherited from Unix's permission bits. The Bell–LaPadula model was later shown to have a subtle flaw: it constrains information flow but not the covert channels through which a high-level process can signal a low-level one by, say, modulating CPU load. Kenneth Biba's 1977 integrity model addressed a different axis, preventing low-integrity data from contaminating high-integrity data5.

These models were not merely academic. They shaped real systems: Honeywell's SCOMP, delivered in 1983, was the first system evaluated at the highest class of the Trusted Computer System Evaluation Criteria (the "Orange Book," 1983), and the later MULTICS and GEMSOS systems implemented multilevel security in production. The enduring lesson was that MAC requires the operating system to enforce policy the user cannot override—a stance that clashed with the Unix tradition of owner-controlled permissions and would resurface decades later in SELinux and AppArmor.

Evaluation criteria and the assurance problem

How do you know a TCB is trustworthy? The Orange Book answered with a ladder of evaluation classes (D through A1), each demanding more evidence: from informal testing at C1, through structured design and covert-channel analysis at B2, to formal top-level specification and verification at A1. The criteria spread internationally—the European ITSEC in 1991, the Canadian CTCPEC, and eventually the Common Criteria (ISO 15408) in 1999, which replaced the national schemes with a framework of protection profiles and evaluation assurance levels (EAL1–EAL7)67.

The assurance problem never fully went away. Formal verification of a whole operating system was attempted—the Kernelized Secure Operating System (KSOS) in the late 1970s, and later the seL4 microkernel, whose functional correctness was machine-checked in 2009—but the cost was enormous, and evaluation schemes were criticized for certifying paperwork as much as code. The deeper issue, articulated by Ken Thompson in his 1984 Turing Award lecture "Reflections on Trusting Trust," is that you cannot verify a compiler with a compiler: trust has to bottom out somewhere, and the TCB concept is really a discipline for being honest about where.

Perimeter defense and the firewall era

Through the 1980s, as machines joined networks, systems security increasingly meant defending a boundary. The first firewalls—packet filters at AT&T Bell Labs in the late 1980s, then application-layer proxies like DEC's SEAL and Marcus Ranum's TIS firewall toolkit in the early 1990s—embodied a simple model: a trusted inside, an untrusted outside, and a chokepoint between them. This perimeter defense was cheap and legible, and it dominated practice for two decades. Its weakness was structural: it assumed the inside was trustworthy, so a single compromised host or a laptop carried past the wall dissolved the boundary. The 1988 Morris worm, which spread through a buffer overflow in fingerd and weak passwords, showed how fast a single flaw could cross a network; the 1999 Melissa and 2000 ILOVEYOU outbreaks showed the same for email.

Host-based monitoring and integrity detection

Inside the perimeter, defenders needed to know when a host had been altered. The idea of checking file integrity against a known-good baseline was formalized by Gene Kim and Eugene Spafford in 1993 with Tripwire, which hashed critical system files and alerted on change. This host-based intrusion detection was joined by log analysis and, later, by kernel-level auditing. The approach's limitation is the baseline problem: on a busy system, files change constantly, and an attacker who compromises the machine can also compromise the checker. The same tension drove the development of antivirus signature scanning (from the late 1980s) and, much later, endpoint detection and response, which watches process behavior rather than file hashes.

Least privilege, vulnerability, and configuration management

Two quieter traditions matured in the 1990s. The first was least privilege in practice: running services as unprivileged users, dropping capabilities, and eventually sandboxing8. The second was the recognition that most compromises exploit misconfiguration and unpatched software rather than novel attacks. Vulnerability scanning tools (SATAN in 1995, Nessus in 1998) and configuration benchmarks turned systems security into a continuous hygiene discipline. The 2000s added automated patch management and, after high-profile breaches traced to default passwords and open services, the idea of a hardened baseline image. This is unglamorous work, but it is where most real incidents are prevented or missed.

Virtualization and workload isolation

The mainframe had virtual machines since the 1960s (IBM's VM/370), but the modern era began when VMware brought x86 virtualization to commodity servers around 1999–2001. Virtualization gave systems security a new isolation primitive: a hypervisor could separate workloads more strongly than an operating system could separate processes, and a compromised guest need not compromise its neighbors. The same logic later produced containers (Linux namespaces and cgroups, popularized by Docker from 2013), which isolate at the operating-system level with weaker but cheaper boundaries. The security question shifted to the hypervisor and container runtime themselves—the new TCB—and to side channels such as the 2017 Meltdown and Spectre speculative-execution flaws, which crossed isolation boundaries that had been assumed solid.

Usable security

A parallel thread asked why users defeat security. In 1999, Alma Whitten and J. D. Tygar published "Why Johnny Can't Encrypt," showing that a well-designed encryption tool was unusable by competent non-experts—and therefore insecure in practice. This founded usable security as an empirical discipline: study what people actually do, measure whether a design helps them do it, and treat unusable security as broken security9. The finding reshaped everything from password policy (the 2017 NIST guidelines abandoned forced complexity and expiry in favor of length and breach checks) to the design of permission prompts and security warnings.

Secure development and the shift-left movement

By the 2000s, the field had learned that security bolted on after deployment is expensive and leaky. Microsoft's Trustworthy Computing memo of 2002, prompted by a series of worms against its products, led to the Security Development Lifecycle (SDL): threat modeling at design time, static analysis and code review during development, penetration testing before release, and incident response after. The secure development lifecycle moved security work earlier—"shift left"—and made it a process rather than a product. It spread through the industry as a set of practices (threat modeling, security champions, bug bounty programs) rather than a single standard.

Zero trust and the present

The perimeter model finally broke under cloud computing and remote work. In 2010, John Kindervag at Forrester argued for zero trust: no implicit trust based on network location; every request authenticated and authorized, every session inspected, least privilege enforced continuously. Google's BeyondCorp (described publicly in 2014) was the landmark implementation, replacing a VPN with per-request access checks1011. Zero trust is less a product than an architecture, and its adoption has been uneven—many organizations run a "zero trust" label over a traditional perimeter—but it has decisively reframed the field's central question from "is this host inside the wall?" to "is this request, from this identity, on this device, authorized for this resource, right now?"

Today the main approaches descend directly from the history above: reference-monitor thinking survives in hypervisors, sandboxes, and hardware roots of trust (TPM, secure enclaves); mandatory access control in SELinux and its kin; evaluation and assurance in the Common Criteria and in formal-verification projects like seL4; integrity and behavior monitoring in endpoint detection; hygiene in vulnerability and configuration management; usable security in every serious design review; and zero trust in the architecture of cloud-native systems121314. The open questions are the old ones in new clothes: how to shrink the TCB when firmware and supply chains are opaque; how to verify that a system enforces its policy when the policy itself is written in natural language and changes constantly; and how to make the secure path the easy path for the people who must walk it.

References

  1. Trusted Computer System Evaluation Criteria (History), Wikipedia
  2. Penetration test (History), Wikipedia
  3. Reference monitor, Wikipedia
  4. Trusted computing base, Wikipedia
  5. Biba Model, Wikipedia
  6. Common Criteria (History), Wikipedia
  7. Evaluation Assurance Level, Wikipedia
  8. Principle of least privilege, Wikipedia
  9. Usable security (introduction, History), Wikipedia
  10. Zero trust architecture (History), Wikipedia
  11. BeyondCorp (introduction, Background), Wikipedia
  12. Computer security (Terminology), Wikipedia
  13. Security-evaluated operating system, Wikipedia
  14. Mandatory access control, Wikipedia

The frameworks of Systems Security

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1972-Present
  2. 1973-Present

    Multilevel Security and Mandatory Access Control

    How can classifications govern who reads and writes what?

    Fly to its tower in Nousfarer

  3. 1983-Present

    Evaluation Criteria and Assurance

    How do we know a system is actually secure?

    Fly to its tower in Nousfarer

  4. 1988-2010

    Firewall

    How do we guard the boundary between networks?

    Fly to its tower in Nousfarer

  5. 1993-Present

    Host-based intrusion detection system

    How do we tell when a host has been altered?

    Fly to its tower in Nousfarer

  6. 1995-Present

    Least Privilege and System Hardening

    How little privilege can each part be given?

    Fly to its tower in Nousfarer

  7. 1995-Present

    Vulnerability and Configuration Management

    Which flaws and misconfigurations leave us exposed?

    Fly to its tower in Nousfarer

  8. 1999-Present

    Usable Security

    Why do people defeat the security we build?

    Fly to its tower in Nousfarer

  9. 2001-Present

    Virtualization and Workload Isolation

    How can workloads be kept apart from one another?

    Fly to its tower in Nousfarer

  10. 2002-Present

    Secure Development Lifecycle

    How do we build security into development from the start?

    Fly to its tower in Nousfarer

  11. 2010-Present

    Zero Trust Architecture

    Should any part of a network be trusted by default?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Reference Monitor and Trusted Computi…Multilevel Security and Mandatory Acc…Evaluation Criteria and AssuranceFirewallHost-based intrusion detection systemLeast Privilege and System HardeningVulnerability and Configuration Manag…Usable SecurityVirtualization and Workload IsolationSecure Development LifecycleZero Trust Architecture

Beyond the field

Where Systems Security's ideas came from in other fields, and what they led to there: the strongest 12 of 49 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Systems Security is a planet in Nousfarer

Its city has a tower for each of these 11 frameworks. Land, walk in, and climb one concept at a time.

Land on Systems Security

Related fields

Fields its ideas are linked to

More in Cybersecurity

See where Systems Security sits on the Map of Human Knowledge