VT-WAM: Visual-Tactile World Action Model for Contact-Rich Manipulation

Noosaga Placements

• Robot Manipulationsubfield95%
  The paper focuses on contact-rich manipulation, a core subfield of robot manipulation, and evaluates on six real-world manipulation tasks.
  VT-WAM: Visual-Tactile World Action Model for Contact-Rich ManipulationAcross six real-world contact-rich manipulation tasks, VT-WAM achieves a 71.67% average success rate
• Learning-Based Manipulationframework95%
  VT-WAM is a learning-based manipulation method that uses flow matching and attention mechanisms to learn policies for contact-rich tasks.
  VT-WAM, a Visual-Tactile World Action Model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching frameworkVT-WAM uses a visual-tactile-action expert architecture
• Robot Learningsubfield90%
  The paper proposes a learning-based method using flow matching, transformers, and attention mechanisms to learn policies from demonstrations.
  VT-WAM jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching frameworkTraining data are collected through human kinesthetic teaching, with 100 expert trajectories for each task
• Imitation Learningframework80%
  The method is trained on expert demonstrations via kinesthetic teaching, which is a form of imitation learning.
  Training data are collected through human kinesthetic teaching, with 100 expert trajectories for each task
• Deep Learningsubfield70%
  The method uses deep learning components including transformers, flow matching, and Mixture-of-Transformers attention.
  Asymmetric Mixture-of-Transformers (MoT) attentionVT-WAM uses pretrained Wan2.2-5B as the visual backbone and uses 1B-scale DiT models for the tactile and action experts
• Transformer Architectureframework70%
  The method uses transformer architecture, specifically Mixture-of-Transformers, for multi-modal attention.
  Asymmetric Mixture-of-Transformers (MoT) attentionVT-WAM uses pretrained Wan2.2-5B as the visual backbone and uses 1B-scale DiT models
• Diffusion Modelsframework60%
  The method uses flow matching, which is a type of diffusion model, for generative modeling of visual, tactile, and action sequences.
  jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching frameworkVT-WAM is trained with a joint flow matching objective over visual, tactile, and action tokens
• Computer Visionsubfield50%
  The method involves visual prediction from wrist camera observations, but the primary focus is on tactile dynamics and manipulation.
  jointly learns future visual prediction, tactile deformation prediction, and action predictionwrist camera observations Ov

Abstract

Contact-rich manipulation requires policies to react to local deformation, pressure, slip, and friction, yet these cues are temporally sparse and often invisible in visual observations. Existing visual-tactile policies usually feed tactile observations directly into action prediction, but rarely model tactile deformation dynamics during action generation. In this paper, we introduce VT-WAM, a Visual-Tactile World Action Model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework. In particular, VT-WAM introduces (1) Asymmetric Mixture-of-Transformers (MoT) attention to bridge a first-frame visual anchor with temporal tactile dynamics, and (2) contact-gated Action-Visual-Tactile Attention Guidance (AVTAG) to encourage action queries to rely on tactile evidence during contact phases. Across six real-world contact-rich manipulation tasks, VT-WAM achieves a 71.67% average success rate, outperforming Fast-WAM by 26.67% and OmniVTLA by 35.84%. Ablations demonstrate that modeling tactile deformation dynamics and guiding contact-phase tactile attention are both important for contact-rich tasks. Project website: https://vt-wam.github.io/.

Paper Context