VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Noosaga Placements

• Robot Learningsubfield95%
  The paper focuses on learning a policy (VLK policy) from synthetic data to predict whole-body kinematic trajectories, which is a core robot learning problem.
  We train a VLK policy that predicts short-horizon whole-body kinematic trajectoriesLearning this mapping requires synchronized egocentric images, language commands, and robot-compatible kinematic trajectories
• Imitation Learningframework90%
  The VLK policy is trained on paired demonstrations (vision, language, kinematics) to predict future trajectories, which is a form of imitation learning from synthetic data.
  We train the policy with an x0-prediction objective in a flow-matching formulationthe generated vision-language-kinematics sequences provide paired supervision for learning a policy
• Robot Manipulationsubfield90%
  The system performs object transport tasks including picking up and placing boxes, which are manipulation tasks.
  object transport requires it to move an object to a commanded locationPick (Floor), Put (Floor), Pick (Surface), Put (Surface)
• Whole-Body and Hierarchical Controlframework85%
  The whole-body tracker is a contact-aware whole-body controller that converts kinematic references into joint-level PD targets for the entire humanoid body.
  a contact-aware whole-body tracker based on SceneBotoutputs joint-level PD targets for all actuated joints
• Robot Controlsubfield85%
  The paper uses a whole-body tracker to convert predicted kinematic trajectories into robot actions, which is a control component.
  A whole-body tracker converts these predictions into actions on the physical humanoidthe tracker follows the reference motion while using the predicted contact labels to stabilize object transport
• Diffusion Modelsframework80%
  The interaction synthesis module uses a conditional diffusion model (DDPM) to generate humanoid-object interaction trajectories.
  We build on the conditional diffusion formulation of prior human-object interaction synthesis methodsThe model is trained as a conditional denoising diffusion model
• Robot Perception And Slamsubfield70%
  The paper uses 3D Gaussian Splatting to reconstruct metric-scale indoor environments, which is a perception and mapping technique.
  Our pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environmentsWe reconstruct indoor environments from Polycam scans
• Vision-Language Modelsframework70%
  The VLK policy takes both egocentric images and language instructions as input, which is a vision-language model approach.
  the VLK policy takes the current egocentric RGB observation ot, task instruction ℓ, and current G1 kinematic state xt as inputWe initialize the VLK policy from a pretrained π0.5
• Computer Visionsubfield60%
  The paper uses egocentric RGB images as input to the policy and renders synthetic egocentric observations, involving computer vision techniques.
  the VLK policy takes the current egocentric RGB observationrenders paired egocentric observations after the fact
• Deep Reinforcement Learningframework50%
  The whole-body tracker is trained with reinforcement learning (RL) to follow reference trajectories, but the paper does not detail RL training for the tracker itself.
  the whole-body tracking policy runs on an RTX 5000 Adathe tracker follows the reference motion

Abstract

Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion. Learning this mapping requires synchronized egocentric images, language commands, and robot-compatible kinematic trajectories, yet no existing data source provides this complete tuple at scale. We address this bottleneck by generating vision-language-kinematics (VLK) supervision synthetically in reconstructed scenes. Our pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environments, synthesizes navigation and object-interaction trajectories using privileged scene information, and renders paired egocentric observations after the fact. We produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts short-horizon whole-body kinematic trajectories. A whole-body tracker converts these predictions into actions on the physical humanoid. We evaluate on the physical Unitree G1 performing navigation and single-object transport, demonstrating that synthesized interactions in reconstructed scenes provide effective supervision for sim-to-real perception-based humanoid loco-manipulation. Project Website: https://vision-language-kinematics.github.io/

Paper Context