Virtual Reality

Virtual reality (VR) is the attempt to put a person inside a computer-generated world: to replace what the eyes, ears, hands, and body would otherwise perceive with synthetic signals, and to make the person feel present in a place that does not physically exist.

13 frameworks, from 1957 to today · Computer Graphics · Computer Science

still a working view ended · each leads to its entry below

Overview

Virtual reality (VR) is the attempt to put a person inside a computer-generated world: to replace what the eyes, ears, hands, and body would otherwise perceive with synthetic signals, and to make the person feel present in a place that does not physically exist. In computer graphics terms, VR is the hardest version of the rendering problem, because the image must be produced not for a passive viewer but for an active participant, fast enough that the world appears to hold still as the head turns. The history of the subfield is therefore a history of two intertwined struggles: building displays and trackers that can fool the senses, and building graphics systems that can keep up with a human body in motion.

Sensorama, Sutherland, and the first head-mounted displays

The earliest working prototypes came from outside academic computing. In the late 1950s and early 1960s Morton Heilig, a cinematographer, built the Sensorama, a coin-operated cabinet that played stereoscopic film with stereo sound, wind, vibration, and smells to a single seated viewer. It was immersive cinema rather than interactive simulation, but it established the multisensory ambition that would recur throughout the field.

The decisive step toward interactive VR was Ivan Sutherland's 1968 "Sword of Damocles" head-mounted display at Harvard and the University of Utah. The display was suspended from the ceiling because it was too heavy for a wearer, and it showed simple wireframe rooms seen from a viewpoint that updated with head position12. Its importance was conceptual: it demonstrated that a computer could render a world that responded to where the user looked, the founding claim of the head-mounted display paradigm. Around the same time, flight simulation was already using wide-field visual systems and motion platforms to train pilots, and the military services became the field's most reliable early patron, funding head-tracked displays and helmet-mounted sights through the 1970s.

The 1980s: gloves, goggles, and the naming of the field

The 1980s turned prototypes into a recognizable research programme. At Bell Laboratories in the early 1980s, Gary Grimes and colleagues built a data glove that measured finger bending, giving the hand a way to act inside a virtual world. At NASA's Ames Research Center, Scott Fisher led the Virtual Interface Environment Workstation project, which combined a wide-angle stereoscopic head-mounted display, head and hand tracking, and 3D audio into a single system. Jaron Lanier's company VPL Research, founded in the mid-1980s, commercialized the DataGlove and the EyePhone headset and popularized the term "virtual reality."13 The era's characteristic claim was that presence—the felt sense of being somewhere else—could be engineered by saturating the senses and coupling them to the body's movements.

The graphics bottleneck was severe. Rendering two eyes at interactive rates with 1980s hardware meant crude geometry and low resolution, and latency between head motion and image update caused nausea. This is where the field's debt to computer graphics became explicit: VR's demands drove work on real-time rendering, stereo display, and low-latency tracking that fed back into graphics generally.

Projection, CAVEs, and the multisensory turn

A competing answer to the head-mounted display emerged in the early 1990s. At the University of Illinois at Chicago, Carolina Cruz-Neira, Daniel Sandin, and Thomas DeFanti built the CAVE, a room whose walls were rear-projection screens showing a stereo image computed for the viewpoint of one tracked viewer wearing shutter glasses. The CAVE traded the headset's intimacy for a wide field of view, low weight, and the ability to share the space with colleagues—an early form of what would later be called social presence. Projection-based immersion became a standard tool for engineering visualization, architectural walkthroughs, and scientific data exploration.

In parallel, researchers pursued multisensory simulation beyond sight and sound: haptic and force-feedback devices such as the PHANToM arm (developed at MIT in the early 1990s) let users feel virtual surfaces, and work on spatial audio, smell, and vestibular cues explored how far the illusion could be pushed. The recurring finding was that presence is not simply a function of pixel count; coherence between what the user does and what the senses report matters more than raw fidelity.

Presence research and the science of "being there"

By the late 1990s a research community had formed around measuring presence itself. The journal Presence: Teleoperators and Virtual Environments, founded in 1992, became its forum, and researchers developed questionnaires and physiological measures to test when users felt transported. The presence research programme reframed VR from a display technology into a psychological phenomenon, and it produced the field's most durable lesson: the goal is not photorealism but perceptual and behavioral plausibility. This insight later shaped everything from rendering priorities to the design of social VR.

The consumer boom and bust

The 1990s saw a wave of consumer VR products—Sega's and Nintendo's headset plans, the Virtuality arcade systems, and a flood of low-cost goggles—backed by a media narrative of imminent mass adoption. Most failed commercially. The displays were heavy, resolution was low, tracking was imprecise, and content was thin. The bust taught the field that the head-mounted display paradigm required components that did not yet exist: cheap, high-resolution microdisplays, low-latency inertial sensors, and graphics chips fast enough to render stereo at frame rates that would not make users sick. The research continued in universities and military labs through the 2000s, but the consumer market went quiet.

Mobile, room-scale, and the second wave

The turnaround came from smartphones. The 2010s put high-resolution displays, tiny inertial measurement units, and powerful mobile graphics into millions of pockets. The Oculus Rift, whose 2012 Kickstarter campaign re-energized the field, and later the HTC Vive and PlayStation VR, combined these components with improved head tracking. A parallel path used phone-based viewers such as Google Cardboard (2014) and Samsung Gear VR to bring cheap VR to a mass audience, and standalone headsets later removed the phone and the cable.

Two technical shifts defined this second wave. First, inside-out and room-scale tracking: instead of external cameras or lighthouses, headsets used onboard cameras and inertial sensors to track their own position and map the room, letting users walk and gesture in a bounded physical space. Second, foveated and perceptually driven rendering: because the eye sees sharply only near the center of gaze, systems could render the periphery at lower resolution and save the compute needed for high frame rates—an idea that became practical once eye-tracking hardware matured. These advances made the head-mounted display paradigm finally comfortable enough for sustained use.

Augmented and mixed reality

Alongside VR, a related line of work sought to overlay synthetic imagery on the real world. Early head-mounted see-through displays appeared in the 1960s and 1970s, and the term "augmented reality" was coined in the early 1990s by Boeing researchers working on aircraft assembly. The 2010s brought marker-based and then markerless tracking on phones, and head-worn systems such as Microsoft's HoloLens (2016) pursued mixed reality—virtual objects anchored in the physical room. Augmented and mixed reality share VR's tracking and rendering problems but add the harder task of registering synthetic content precisely against a real, moving scene.

Neural rendering and the present

The most recent turn comes from machine learning. Neural rendering methods learn to synthesize images from data—photographs, scans, or captured light fields—rather than describing scenes by hand. Neural radiance fields (NeRF), introduced in 2020, and later 3D Gaussian splatting made it possible to reconstruct photorealistic, view-dependent scenes from a handful of photographs, and to render them at interactive rates. For VR this is significant because it lowers the cost of building believable virtual environments and offers new ways to capture real places for immersive replay. The open questions are the ones the field has carried since Sutherland: latency, comfort, field of view, and the stubborn gap between a convincing picture and a convincing sense of being there.

References

  1. Virtual reality (20th century, Etymology, 1970–1990), Wikipedia
  2. Head-mounted display (History), Wikipedia
  3. VPL Research, Wikipedia

The frameworks of Virtual Reality

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1957-1970

    Multisensory immersion

    Can a machine surround a viewer with convincing synthetic senses?

    Fly to its tower in Nousfarer

  2. 1960-Present

    Flight simulation and military training systems

    How can simulation train people for dangerous real tasks?

    Fly to its tower in Nousfarer

  3. 1968-Present

    Head-mounted display

    Can a display replace what the eyes see?

    Fly to its tower in Nousfarer

  4. 1982-Present

    Wired glove

    How can a hand act inside a virtual world?

    Fly to its tower in Nousfarer

  5. 1990-Present

    Augmented reality

    Can synthetic imagery be laid over the real world?

    Fly to its tower in Nousfarer

  6. 1991-1999

    Consumer VR product wave

    Can immersive headsets be sold to ordinary people?

    Fly to its tower in Nousfarer

  7. 1992-Present

    Cave automatic virtual environment

    Can a room of screens immerse a whole viewer?

    Fly to its tower in Nousfarer

  8. 1992-Present

    Presence

    When does a person feel present in a virtual place?

    Fly to its tower in Nousfarer

  9. 1993-Present

    Haptic technology

    How can a machine let a person feel virtual surfaces?

    Fly to its tower in Nousfarer

  10. 2014-Present

    Mobile and standalone VR

    Can a phone or standalone device deliver VR?

    Fly to its tower in Nousfarer

  11. 2015-Present

    Room-scale and inside-out tracking

    How can a headset know where it is in a room?

    Fly to its tower in Nousfarer

  12. 2016-Present

    Foveated rendering

    How can VR render fast enough without wasting pixels?

    Fly to its tower in Nousfarer

  13. 2020-Present

    Neural rendering

    Can learned models synthesize believable views from data?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Multisensory immersionFlight simulation and military traini…Head-mounted displayWired gloveAugmented realityConsumer VR product waveCave automatic virtual environmentPresenceHaptic technologyMobile and standalone VRRoom-scale and inside-out trackingFoveated renderingNeural rendering

Beyond the field

Where Virtual Reality's ideas came from in other fields, and what they led to there: the strongest 12 of 33 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Virtual Reality is a planet in Nousfarer

Its city has a tower for each of these 13 frameworks. Land, walk in, and climb one concept at a time.

Land on Virtual Reality

Related fields

Fields its ideas are linked to

More in Computer Graphics

See where Virtual Reality sits on the Map of Human Knowledge