- Atlas
- Computer Science
- Embodied Interaction
Embodied Interaction
Embodied interaction is the branch of human–computer interaction (HCI) that treats meaning, skill, and knowledge as arising from bodies moving and acting in the world, rather than from minds manipulating symbols in isolation.
8 frameworks, from 1982 to today · Human Computer Interaction · Computer Science
Overview
Embodied interaction is the branch of human–computer interaction (HCI) that treats meaning, skill, and knowledge as arising from bodies moving and acting in the world, rather than from minds manipulating symbols in isolation. Where classical HCI asked how an interface could fit a user's mental model, embodied interaction asks how a system can fit a user's hands, habits, senses, and situations. What is at stake is a fundamental question about where the human being sits in design: is the person a reasoning processor receiving information through a screen, or a bodily agent who knows the world through engaged action? The history below follows that question through its several waves—direct manipulation, the turn to the situated and the tangible, the spread of gestural and surface computing, and the recent move toward designing for the felt, moving body itself.
Direct Manipulation: The Body Enters Through the Hand
The earliest practical ancestor of embodied interaction was direct manipulation, a term coined by Ben Shneiderman in the early 1980s1. The problem it addressed was the command-line interface, where users had to remember and type arcane syntax and could not see the consequences of their actions until after the fact. Shneiderman observed that a different style of interface—visible objects on a screen, manipulated by pointing, with effects shown immediately and actions reversible—was both easier to learn and more satisfying. His canonical examples were the Xerox Star and the Apple Macintosh desktop, where dragging a file icon into a folder icon visually "moved" the document. The idea was that the user acts on representations as if they were the things themselves, closing the gap between intention and effect.
Direct manipulation was assimilationist rather than radical: it kept the user outside the system, looking at a screen and operating through a pointer. But it introduced a bodily metaphor—the hand reaching for and grasping an object—that later researchers would push much further. Its vocabulary of "grasping," "dragging," and "directness" set the terms of the debate that followed.
Situated Action and the Critique of the Plan
In the late 1980s and early 1990s, a group of researchers drawing on sociology and anthropology challenged the cognitive assumptions beneath both direct manipulation and the prevailing information-processing model. Lucy Suchman's Plans and Situated Actions (1987) studied people using a photocopier's help system and argued that human action is not the execution of pre-formed plans but an improvisation within a concrete situation2. Her well-known example contrasted the orderly instructions the machine offered with the messy, context-sensitive ways users actually figured out what to do. The implication for design was uncomfortable: no plan or interface could fully anticipate what a person would do, because the situation always supplied resources the designer had not foreseen.
Alongside this, researchers in computer-supported cooperative work (CSCW) and ethnomethodology—such as those studying air traffic control rooms and control centers—showed that work is organized through talk, gesture, and shared awareness of the physical setting. The term "situated action" named this view: action is always embedded in a here and now, and meaning is produced locally rather than stored in a head. For HCI this meant that designs had to be evaluated in real settings, not just in controlled experiments, and that the body's placement in a shared space mattered.
Embodied Cognition and the Theoretical Reframing
The intellectual scaffolding for embodied interaction came from cognitive science and philosophy. Work on embodied cognition argued that cognitive processes are shaped by, and sometimes constituted by, the body and its sensorimotor capacities—a view often traced to the phenomenology of Maurice Merleau-Ponty and to later experimental work on how bodily states influence thought. In HCI, Paul Dourish's Where the Action Is: The Foundations of Embodied Interaction (2001) gave the field its name and a program. Dourish drew on phenomenology and on tangible computing to argue that interaction should be understood as an embodied phenomenon: users engage with systems through their physical presence and practices, and designers should build technologies that fit those practices rather than abstract task models.
This was a theoretical reframing as much as a design technique. It replaced the image of a user mentally representing a system with the image of a person acting within a world of physical and social affordances. The term "affordance" itself, borrowed from James J. Gibson's ecological psychology, entered the design vocabulary around this time to describe what an environment offers an animal that can perceive and act3.
Tangible Computing and the Return of the Physical
If direct manipulation put a metaphor of touch on a screen, tangible computing put computation into graspable physical objects. The problem it addressed was the friction of treating everything as pixels: some tasks—arranging spatial data, doing design, playing—seem more natural with things you can pick up and move. Hiroshi Ishii and Brygg Ullmer's "Tangible Bits" work at the MIT Media Lab in the late 1990s proposed marrying digital information to physical artifacts and ambient displays, so that the "bits" could be handled like "atoms."4 A well-known example is the marble answering machine or the physical icons—"phicons"—used to represent and manipulate digital content4. The claim was that physically manipulating tokens exploits human dexterity, spatial memory, and social gesturing that a mouse and screen flatten.
Tangible computing also connected HCI to architecture and product design, producing installations, augmented tabletops, and interactive furniture. Its tension with embodied interaction was productive: tangible systems raised the question of whether "embodiment" is best achieved by bringing computation into the physical world or by bringing the body's movement into computational sensing.
Gesture, Movement, and the Moving Body as Input
From the late 1990s onward, vision-based sensing and motion capture made the body itself an input device5. Early work in gesture-based interaction recognized that humans gesture naturally while talking and thinking, and that a system which could read those gestures might feel more fluid than one requiring a keyboard. William Buxton's research in the 1980s on "there and then" versus "here and now" input, and later work on free-hand and pen gestures, established that different input modes suit different kinds of tasks.
The mass-market arrival of gesture and movement interfaces—the Nintendo Wii in 2006 with its motion-sensing remote, Microsoft Kinect in 2010 with full-body tracking—brought these ideas to millions. Kinect in particular sparked a wave of research on full-body interaction, rehabilitation, and playful movement. But the field's designers quickly learned a lesson familiar from direct manipulation: movement is not automatically intuitive. Systems like the early wave of "minority report"–style free-hand interfaces often exhausted users and lacked the precision of a physical control. This led to "movement-based interaction design," which studies how whole-body movement can be used meaningfully—for exertion games, for learning, and for expressive performance—rather than as a novelty. Work on dance and interactive performance, and on "embodied metaphors" for learning (for example, having children walk along a number line to learn arithmetic), treated movement as a medium of thought, not merely a command.
Surface, Tabletop, and the Social Body
In parallel, multi-touch and surface interaction developed from research prototypes in the 1980s into everyday technology with the iPhone (2007) and large interactive tables6. Early systems like the Digital Desk, and later the DiamondTouch and Microsoft Surface tables, explored direct touching with fingers and multiple simultaneous users. The significance for embodied interaction was social and spatial: horizontal surfaces invite people to stand around them, point, and collaborate, restoring some of the qualities of a shared table. Multi-touch also demonstrated that a rich vocabulary of touch—pinching, swiping, two-finger rotation—could be learned quickly and remembered, another instance of the body's skill being harnessed by design.
Surface interaction connected directly back to tangible computing, since tabletop systems often combine physical tokens with touch, and back to situated action, since the tabletop's effectiveness depends on who stands where and what they can see.
Somaesthetics and Designing for the Felt Body
The most recent strand, emerging in the 2010s, pushes embodied interaction inward: toward the felt, proprioceptive, emotional body of the user, and toward the designer's own body as a source of knowledge. Somaesthetic design, associated with Kristina Höök and colleagues (her book Designing with the Body: Somaesthetic Interaction Design, 2018), draws on the philosopher Richard Shusterman's notion of somaesthetics—the idea that bodily experience can be cultivated and refined78. The design practice involves designers attending to their own bodily sensations—breath, balance, tension—and using those first-person experiences to imagine and craft interactions. Concrete examples include systems that guide breathing, wearable designs that encourage movement, and training methods that ask designers to move, feel, and reflect before prototyping.
Somaesthetic design claimed that earlier embodied approaches, for all their talk of the body, often treated it as a sensor or a motor system rather than as a locus of lived experience. Its compensation was methodological: first-person, experiential, and sometimes artistic methods joined ethnographic and experimental ones.
The Present: Where Embodied Interaction Lives Now
Today embodied interaction is less a single method than a broad orientation cutting across HCI. Its direct descent runs through direct manipulation's descendants (touchscreen interfaces with physical-feeling gestures), tangible and surface computing (interactive tables and physical-digital hybrids), and gesture and movement systems (virtual reality controllers that track hands and heads, and full-body experiences). Virtual and augmented reality have become important arenas, because head and hand tracking make the body the primary instrument of interaction and reintroduce the problem of designing for spatial, proprioceptive understanding rather than for a flat screen910.
The open questions are continuous with the history. One is how to design movement-based interaction that is meaningful rather than merely novel—how to know when a gesture is intuitive and when it is an arbitrary code. Another is how first-person, felt experience can be evaluated rigorously without reducing it to a task metric. A third is how embodied ideas travel across cultures and abilities, since what a body knows is not universal; accessible and inclusive design increasingly asks whose bodies embodied interaction is really designed for. The through-line remains the one that direct manipulation first opened: that the hand, the moving body, and the situation of action are not peripheral to computing but are where meaning is made.
References
- Direct manipulation interface, Wikipedia
- Situated cognition (History), Wikipedia
- Affordance, Wikipedia
- Tangible user interface (introduction, History), Wikipedia
- Motion capture, Wikipedia
- Multi-touch, Wikipedia
- Kristina Höök, Wikipedia
- Somaesthetics (introduction, Origin and development), Wikipedia
- Human–computer interaction (Current research), Wikipedia
- Virtual reality, Wikipedia
The frameworks of Embodied Interaction
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1982-Present
Direct manipulation interface
How should users manipulate on-screen objects directly?
- 1987-Present
- 1990-Present
- 1991-Present
- 1997-Present
Tangible user interface
How can physical objects carry digital information?
- 2000-Present
Movement-Based Interaction Design
How can whole-body movement be meaningful in design?
- 2003-Present
- 2011-Present
Somaesthetic Interaction Design
How can design address the felt, moving body?
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Multi-touch influenced by Direct manipulation interface
Multi-touch surfaces extended direct manipulation's directness to finger touch, replacing the pointer with the hand itself while keeping the immediate-manipulation model.
Carried by Pierre Wellner; Digital Desk (1991).
Tangible user interface influenced by Direct manipulation interface
Tangible Bits extended direct manipulation's principle of acting on visible representations to physical tokens, keeping the grasp-and-manipulate metaphor but moving it off the screen.
Carried by Hiroshi Ishii; Brygg Ullmer; Tangible Bits (1997).
Movement-Based Interaction Design influenced by Embodied Cognition
Movement-based interaction design drew on embodied cognition to treat whole-body movement as a medium of thought and learning, not merely a command signal.
Somaesthetic Interaction Design influenced by Embodied Cognition
Somaesthetic interaction design extended embodied cognition inward to the felt, proprioceptive body and to the designer's own bodily experience as a source of knowledge.
Gesture-Based Interaction influenced by Multi-touch
Multi-touch established a vocabulary of finger gestures (pinch, swipe, rotate) that gesture-based interaction generalized to free-hand and whole-body gestures.
Somaesthetic Interaction Design influenced by Movement-Based Interaction Design
Somaesthetic design took from movement-based interaction the focus on the moving body, adding first-person felt experience and somatic awareness.
Beyond the field
Where Embodied Interaction's ideas came from in other fields, and what they led to there: the strongest 12 of 31 links.
Direct manipulation interface influenced by Interactive graphics and direct manipulation in Computer Graphics
Shneiderman built direct manipulation explicitly on the interactive graphics tradition of Sutherland's Sketchpad and successors, taking the idea of manipulating visible objects on screen with immediate feedback.
Carried by Ivan Sutherland; Sketchpad (1963); Ben Shneiderman.
Situated Action influenced Embodied Interaction in Cognitive Hci
Dourish's embodied interaction explicitly incorporates situated action as one of its two foundations, alongside phenomenology.
Carried by Paul Dourish; Lucy Suchman.
Situated Action influenced Embodied Interaction in Human Computer Interaction
Situated action, from Suchman's Plans and Situated Actions (1987), supplied Dourish with the claim that action is contingent and context-bound; he treated it as one of the two roots of embodied interaction.
Carried by Lucy Suchman; Plans and Situated Actions (1987); Paul Dourish.
Embodied Cognition influenced by Phenomenological Embodiment in Embodied Cognition
Merleau-Ponty's phenomenology of the lived body, especially the 1945 Phenomenology of Perception, supplied the central concept of pre-reflective bodily engagement that embodied cognition took up and generalised.
Carried by Maurice Merleau-Ponty; Phenomenology of Perception (1945).
Embodied Cognition influenced Embodied Interaction in Cognitive Hci
Dourish's embodied interaction was built directly on embodied cognition, taking its rejection of the separation of mind from body and world as the foundation for interaction design.
Carried by Paul Dourish.
Embodied Cognition influenced Embodied Interaction in Human Computer Interaction
Dourish built Embodied Interaction explicitly on embodied cognition, drawing on phenomenology and on the cognitive-science critique of representationalism to argue that meaning arises from engaged practice rather than internal models.
Carried by Paul Dourish; Where the Action Is (2001).
Tangible user interface influenced by Direct manipulation interface in Human Computer Interaction
Tangible user interfaces were framed by Ishii and Ullmer as extending direct manipulation from screen pixels to physical objects, keeping the directness of grasping and moving while changing the medium.
Carried by Hiroshi Ishii; Brygg Ullmer.
Tangible user interface influenced by Direct manipulation interface in Interaction Design
Tangible interfaces took the direct manipulation principle of acting on represented objects with immediate feedback and applied it to graspable physical tokens.
Carried by Hiroshi Ishii.
Gesture-Based Interaction influenced by Embodied cognition in Cognitive Psychology
Embodied cognition, especially the phenomenological and Merleau-Pontian strand, gave embodied interaction its central claim that meaning and action arise from bodily engagement. Dourish's 2001 book drew directly on this literature to reframe HCI.
Carried by Paul Dourish; Maurice Merleau-Ponty.
Somaesthetic Interaction Design influenced by Embodied Interaction in Cognitive Hci
Embodied interaction, especially Dourish's phenomenological account, provided the theoretical foundation somaesthetic interaction design extended from interaction-in-the-world to felt somatic experience. Höök built explicitly on this body of work.
Carried by Kristina Höök; Paul Dourish.
Somaesthetic Interaction Design influenced by Somatic Dance Pedagogy in Dance Somatics Theory
Somatic dance pedagogy, particularly Feldenkrais and related body-awareness practices, supplied the movement-awareness exercises and first-person methods that somaesthetic interaction design adapted for design workshops. Höök drew directly on these pedagogical traditions.
Carried by Kristina Höök; Moshe Feldenkrais.
Direct manipulation interface influenced Tangible and Embodied Interaction in Interaction Design
Direct manipulation's ideal of engaging with objects rather than commands was taken up by tangible interaction, which made the manipulated objects physical.
Carried by Ben Shneiderman.

Embodied Interaction is a planet in Nousfarer
Its city has a tower for each of these 8 frameworks. Land, walk in, and climb one concept at a time.
Land on Embodied InteractionRelated fields
Fields its ideas are linked to
- Interaction Design 9 links
- Human Computer Interaction 8 links
- Dance Somatics Theory 3 links
- Interaction Design Theory 3 links
- Cognitive Hci 3 links
- Dance And Technology 1 link
- Embodied Cognition 1 link
- Computer Graphics 1 link
- Human Factors 1 link
- Cognitive Psychology 1 link
More in Human Computer Interaction
- Accessibility
- Cognitive Hci
- Critical Hci
- Cscw
- Human Computer Interaction
- Interaction Design
- Participatory Design
- Ubiquitous Computing
See where Embodied Interaction sits on the Map of Human Knowledge