Embodied Interaction

Embodied interaction is the branch of human–computer interaction (HCI) that treats meaning, skill, and knowledge as arising from bodies moving and acting in the world, rather than from minds manipulating symbols in isolation.

8 frameworks, from 1982 to today · Human Computer Interaction · Computer Science

still a working view ended · each leads to its entry below

Overview

Embodied interaction is the branch of human–computer interaction (HCI) that treats meaning, skill, and knowledge as arising from bodies moving and acting in the world, rather than from minds manipulating symbols in isolation. Where classical HCI asked how an interface could fit a user's mental model, embodied interaction asks how a system can fit a user's hands, habits, senses, and situations. What is at stake is a fundamental question about where the human being sits in design: is the person a reasoning processor receiving information through a screen, or a bodily agent who knows the world through engaged action? The history below follows that question through its several waves—direct manipulation, the turn to the situated and the tangible, the spread of gestural and surface computing, and the recent move toward designing for the felt, moving body itself.

Direct Manipulation: The Body Enters Through the Hand

The earliest practical ancestor of embodied interaction was direct manipulation, a term coined by Ben Shneiderman in the early 1980s1. The problem it addressed was the command-line interface, where users had to remember and type arcane syntax and could not see the consequences of their actions until after the fact. Shneiderman observed that a different style of interface—visible objects on a screen, manipulated by pointing, with effects shown immediately and actions reversible—was both easier to learn and more satisfying. His canonical examples were the Xerox Star and the Apple Macintosh desktop, where dragging a file icon into a folder icon visually "moved" the document. The idea was that the user acts on representations as if they were the things themselves, closing the gap between intention and effect.

Direct manipulation was assimilationist rather than radical: it kept the user outside the system, looking at a screen and operating through a pointer. But it introduced a bodily metaphor—the hand reaching for and grasping an object—that later researchers would push much further. Its vocabulary of "grasping," "dragging," and "directness" set the terms of the debate that followed.

Situated Action and the Critique of the Plan

In the late 1980s and early 1990s, a group of researchers drawing on sociology and anthropology challenged the cognitive assumptions beneath both direct manipulation and the prevailing information-processing model. Lucy Suchman's Plans and Situated Actions (1987) studied people using a photocopier's help system and argued that human action is not the execution of pre-formed plans but an improvisation within a concrete situation2. Her well-known example contrasted the orderly instructions the machine offered with the messy, context-sensitive ways users actually figured out what to do. The implication for design was uncomfortable: no plan or interface could fully anticipate what a person would do, because the situation always supplied resources the designer had not foreseen.

Alongside this, researchers in computer-supported cooperative work (CSCW) and ethnomethodology—such as those studying air traffic control rooms and control centers—showed that work is organized through talk, gesture, and shared awareness of the physical setting. The term "situated action" named this view: action is always embedded in a here and now, and meaning is produced locally rather than stored in a head. For HCI this meant that designs had to be evaluated in real settings, not just in controlled experiments, and that the body's placement in a shared space mattered.

Embodied Cognition and the Theoretical Reframing

The intellectual scaffolding for embodied interaction came from cognitive science and philosophy. Work on embodied cognition argued that cognitive processes are shaped by, and sometimes constituted by, the body and its sensorimotor capacities—a view often traced to the phenomenology of Maurice Merleau-Ponty and to later experimental work on how bodily states influence thought. In HCI, Paul Dourish's Where the Action Is: The Foundations of Embodied Interaction (2001) gave the field its name and a program. Dourish drew on phenomenology and on tangible computing to argue that interaction should be understood as an embodied phenomenon: users engage with systems through their physical presence and practices, and designers should build technologies that fit those practices rather than abstract task models.

This was a theoretical reframing as much as a design technique. It replaced the image of a user mentally representing a system with the image of a person acting within a world of physical and social affordances. The term "affordance" itself, borrowed from James J. Gibson's ecological psychology, entered the design vocabulary around this time to describe what an environment offers an animal that can perceive and act3.

Tangible Computing and the Return of the Physical

If direct manipulation put a metaphor of touch on a screen, tangible computing put computation into graspable physical objects. The problem it addressed was the friction of treating everything as pixels: some tasks—arranging spatial data, doing design, playing—seem more natural with things you can pick up and move. Hiroshi Ishii and Brygg Ullmer's "Tangible Bits" work at the MIT Media Lab in the late 1990s proposed marrying digital information to physical artifacts and ambient displays, so that the "bits" could be handled like "atoms."4 A well-known example is the marble answering machine or the physical icons—"phicons"—used to represent and manipulate digital content4. The claim was that physically manipulating tokens exploits human dexterity, spatial memory, and social gesturing that a mouse and screen flatten.

Tangible computing also connected HCI to architecture and product design, producing installations, augmented tabletops, and interactive furniture. Its tension with embodied interaction was productive: tangible systems raised the question of whether "embodiment" is best achieved by bringing computation into the physical world or by bringing the body's movement into computational sensing.

Gesture, Movement, and the Moving Body as Input

From the late 1990s onward, vision-based sensing and motion capture made the body itself an input device5. Early work in gesture-based interaction recognized that humans gesture naturally while talking and thinking, and that a system which could read those gestures might feel more fluid than one requiring a keyboard. William Buxton's research in the 1980s on "there and then" versus "here and now" input, and later work on free-hand and pen gestures, established that different input modes suit different kinds of tasks.

The mass-market arrival of gesture and movement interfaces—the Nintendo Wii in 2006 with its motion-sensing remote, Microsoft Kinect in 2010 with full-body tracking—brought these ideas to millions. Kinect in particular sparked a wave of research on full-body interaction, rehabilitation, and playful movement. But the field's designers quickly learned a lesson familiar from direct manipulation: movement is not automatically intuitive. Systems like the early wave of "minority report"–style free-hand interfaces often exhausted users and lacked the precision of a physical control. This led to "movement-based interaction design," which studies how whole-body movement can be used meaningfully—for exertion games, for learning, and for expressive performance—rather than as a novelty. Work on dance and interactive performance, and on "embodied metaphors" for learning (for example, having children walk along a number line to learn arithmetic), treated movement as a medium of thought, not merely a command.

Surface, Tabletop, and the Social Body

In parallel, multi-touch and surface interaction developed from research prototypes in the 1980s into everyday technology with the iPhone (2007) and large interactive tables6. Early systems like the Digital Desk, and later the DiamondTouch and Microsoft Surface tables, explored direct touching with fingers and multiple simultaneous users. The significance for embodied interaction was social and spatial: horizontal surfaces invite people to stand around them, point, and collaborate, restoring some of the qualities of a shared table. Multi-touch also demonstrated that a rich vocabulary of touch—pinching, swiping, two-finger rotation—could be learned quickly and remembered, another instance of the body's skill being harnessed by design.

Surface interaction connected directly back to tangible computing, since tabletop systems often combine physical tokens with touch, and back to situated action, since the tabletop's effectiveness depends on who stands where and what they can see.

Somaesthetics and Designing for the Felt Body

The most recent strand, emerging in the 2010s, pushes embodied interaction inward: toward the felt, proprioceptive, emotional body of the user, and toward the designer's own body as a source of knowledge. Somaesthetic design, associated with Kristina Höök and colleagues (her book Designing with the Body: Somaesthetic Interaction Design, 2018), draws on the philosopher Richard Shusterman's notion of somaesthetics—the idea that bodily experience can be cultivated and refined78. The design practice involves designers attending to their own bodily sensations—breath, balance, tension—and using those first-person experiences to imagine and craft interactions. Concrete examples include systems that guide breathing, wearable designs that encourage movement, and training methods that ask designers to move, feel, and reflect before prototyping.

Somaesthetic design claimed that earlier embodied approaches, for all their talk of the body, often treated it as a sensor or a motor system rather than as a locus of lived experience. Its compensation was methodological: first-person, experiential, and sometimes artistic methods joined ethnographic and experimental ones.

The Present: Where Embodied Interaction Lives Now

Today embodied interaction is less a single method than a broad orientation cutting across HCI. Its direct descent runs through direct manipulation's descendants (touchscreen interfaces with physical-feeling gestures), tangible and surface computing (interactive tables and physical-digital hybrids), and gesture and movement systems (virtual reality controllers that track hands and heads, and full-body experiences). Virtual and augmented reality have become important arenas, because head and hand tracking make the body the primary instrument of interaction and reintroduce the problem of designing for spatial, proprioceptive understanding rather than for a flat screen910.

The open questions are continuous with the history. One is how to design movement-based interaction that is meaningful rather than merely novel—how to know when a gesture is intuitive and when it is an arbitrary code. Another is how first-person, felt experience can be evaluated rigorously without reducing it to a task metric. A third is how embodied ideas travel across cultures and abilities, since what a body knows is not universal; accessible and inclusive design increasingly asks whose bodies embodied interaction is really designed for. The through-line remains the one that direct manipulation first opened: that the hand, the moving body, and the situation of action are not peripheral to computing but are where meaning is made.

References

  1. Direct manipulation interface, Wikipedia
  2. Situated cognition (History), Wikipedia
  3. Affordance, Wikipedia
  4. Tangible user interface (introduction, History), Wikipedia
  5. Motion capture, Wikipedia
  6. Multi-touch, Wikipedia
  7. Kristina Höök, Wikipedia
  8. Somaesthetics (introduction, Origin and development), Wikipedia
  9. Human–computer interaction (Current research), Wikipedia
  10. Virtual reality, Wikipedia

The frameworks of Embodied Interaction

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1982-Present

    Direct manipulation interface

    How should users manipulate on-screen objects directly?

    Fly to its tower in Nousfarer

  2. 1987-Present

    Situated Action

    How do people actually act in real situations?

    Fly to its tower in Nousfarer

  3. 1990-Present

    Embodied Cognition

    How does the body shape thought and interaction?

    Fly to its tower in Nousfarer

  4. 1991-Present

    Multi-touch

    How can surfaces sense multiple fingers at once?

    Fly to its tower in Nousfarer

  5. 1997-Present

    Tangible user interface

    How can physical objects carry digital information?

    Fly to its tower in Nousfarer

  6. 2000-Present

    Movement-Based Interaction Design

    How can whole-body movement be meaningful in design?

    Fly to its tower in Nousfarer

  7. 2003-Present

    Gesture-Based Interaction

    How can systems read human gestures as input?

    Fly to its tower in Nousfarer

  8. 2011-Present

    Somaesthetic Interaction Design

    How can design address the felt, moving body?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Direct manipulation interfaceSituated ActionEmbodied CognitionMulti-touchTangible user interfaceMovement-Based Interaction DesignGesture-Based InteractionSomaesthetic Interaction Design

Beyond the field

Where Embodied Interaction's ideas came from in other fields, and what they led to there: the strongest 12 of 31 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Embodied Interaction is a planet in Nousfarer

Its city has a tower for each of these 8 frameworks. Land, walk in, and climb one concept at a time.

Land on Embodied Interaction

Related fields

Fields its ideas are linked to

More in Human Computer Interaction

See where Embodied Interaction sits on the Map of Human Knowledge