VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity

Noosaga Placements

• Deep Reinforcement Learningframework95%
  The paper directly uses PPO (a deep RL algorithm) to train manipulation policies in simulation.
  We train PPO policies entirely in simulation.We train consistent policy architectures across all five tasks in MuJoCo simulation [45] with PPO [10]
• Robot Manipulationsubfield95%
  The paper addresses dexterous manipulation tasks such as regrasping, in-hand reorientation, and peg insertion, and proposes a method specifically for contact-rich manipulation on a dexterous hand.
  VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared physical representation of contact and slip.Across five contact-rich tasks spanning regrasping, in-hand reorientation, and insertion...
• Learning-Based Manipulationframework95%
  The paper develops a learning-based approach for dexterous manipulation, employing both a learned tactile estimator and a learned policy.
  We propose VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning...The learned policies transfer to a physical dexterous hand-arm platform...
• Robot Learningsubfield90%
  The core method is a sim-to-real RL framework: policies are trained via PPO in simulation on a shared tactile representation, and then transferred to a real robot.
  VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared physical representation of contact and slip.We train PPO policies entirely in simulation.
• Robot Perception And Slamsubfield60%
  The paper builds a tactile estimator that maps raw microphone signals to a low-dimensional tactile representation, which can be viewed as a perception module that extracts task-relevant state information from high-bandwidth sensors.
  A tactile estimator learns to predict contact and slip from real microphone waveforms...The real-world perception problem is to learn a mapping fθ : xt−w:t 7→ zt from vibroacoustic signals to tactile representation.
• Robot Controlsubfield50%
  The output of the policy is continuous actions for arm and hand joints, and the work demonstrates reactive control behaviors such as slip compensation during insertion.
  It outputs a continuous action at for the arm and hand joints.where the continuous slip-magnitude channel proves the most informative observation for sustained reactive control.

Abstract

Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones offer a compact and high-bandwidth way to sense these interactions, but the resulting vibro-acoustic signals are difficult to simulate faithfully enough for end-to-end sim-to-real policy learning on dexterous robot hands. We propose VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared physical representation of contact and slip. In the real world, we embed piezoelectric microphones into a dexterous robot hand and collect vibro-acoustic data through teleoperation, then replay the recordings in a calibrated digital clone to automatically label per-finger contact and slip. A tactile estimator learns to predict contact and slip from real microphone waveforms, while manipulation policies are trained in simulation on the same representation computed directly from simulated contacts. This decoupling lets policies exploit rapid tactile feedback without simulating raw audio. Across five contact-rich tasks spanning regrasping, in-hand reorientation, and insertion, VibeAct consistently outperforms a proprioception-and-point-cloud baseline in simulation, with the largest gains on tasks requiring sustained reactive control, where the continuous slip-magnitude channel proves the most informative observation. The learned policies transfer to a physical dexterous hand-arm platform, improving success rates on deployed tasks. Project videos and additional details are at https://vibeact.github.io/.

Paper Context