Native Video-Action Pretraining for Generalizable Robot Control

Noosaga Placements

• Robot Learningsubfield95%
  The paper focuses on learning a video-action foundation model for robot control, which is a core topic in robot learning. The methods involve pretraining, few-shot adaptation, and imitation learning.
  We present LingBot-VA 2.0, a video-action foundation model built from the ground up for embodiment.Real-world deployment validates LingBot-VA 2.0 as a robust foundation model, as evidenced by its few-shot generalization across complex manipulation tasks.
• Imitation Learningframework90%
  The paper uses imitation learning (behavioral cloning) to train the policy from demonstrations, as evidenced by the supervised fine-tuning on robot demonstration data.
  For each task, we collect 20 teleoperated demonstrations and train a single generalist policy with multi-task supervised fine-tuning
• Learning-Based Robot Controlframework90%
  The paper presents a learning-based robot control method, where the control policy is learned from data using a video-action foundation model.
  We present LingBot-VA 2.0, a video-action foundation model built from the ground up for embodiment.Real-time closed-loop control is realized through an enhanced asynchronous inference scheme
• Robot Controlsubfield90%
  The paper directly addresses robot control, specifically real-time closed-loop control, and introduces a novel asynchronous inference scheme (Foresight Reasoning) for high-frequency control.
  Real-time closed-loop control is realized through an enhanced asynchronous inference scheme, which predicts future latents in parallel with action execution while re-grounding each rollout on the latest observation via learned forward dynamics.LingBot-VA 2.0 to perform real-time closed-loop control on real robots with a peak asynchronous execution frequency of 225 Hz.
• Robot Manipulationsubfield85%
  The paper evaluates on complex manipulation tasks, including bimanual manipulation, and the model is designed for generalist robot manipulation.
  Real-world deployment validates LingBot-VA 2.0 as a robust foundation model, as evidenced by its few-shot generalization across complex manipulation tasks.We evaluate real-world manipulation on an in-house robot benchmark covering a diverse set of everyday manipulation tasks.
• Model-Based Reinforcement Learningframework85%
  The paper uses a learned forward dynamics model (FDM) as part of the Foresight Reasoning scheme, which is a form of model-based control. The model predicts future visual latents, which is a world model.
  Foresight Reasoning, an asynchronous inference scheme in which future visual latents are predicted in parallel with action execution.The policy’s video expert then runs a forward-dynamics pass, performed online by the policy itself rather than by the tokenizer’s frozen FDM fψ , that imagines the visual outcome of the current chunk
• Learning-Based Manipulationframework85%
  The paper focuses on learning manipulation skills from demonstrations, which is a core aspect of learning-based manipulation.
  Real-world deployment validates LingBot-VA 2.0 as a robust foundation model, as evidenced by its few-shot generalization across complex manipulation tasks.
• Deep Learningsubfield80%
  The paper heavily relies on deep learning techniques, including diffusion transformers, mixture-of-experts, and representation learning, and introduces a new pretraining paradigm.
  we adopt a causal pretraining paradigm, training from scratch to circumvent the catastrophic forgetting that frequently occurs when adapting bidirectional architectures.our model employs a sparse MoE backbone, expanding model capacity without compromising efficiency.
• Vision Transformers and Foundation Modelsframework80%
  The paper uses a vision transformer (DiT) and a vision-language model (VLM) for the planner, which are foundation models in computer vision.
  The second stage is a causal diffusion transformer that predicts future visual latents together with the latent actions connecting themWe implement the planner as a pretrained vision-language model (VLM) that is fine-tuned to produce structured subtask context for the policy.
• Computer Visionsubfield70%
  The paper uses a semantic visual-action tokenizer that aligns visual representations with a vision foundation model, and the model predicts future visual latents, which are core computer vision tasks.
  we introduce a semantic visual-action tokenizer, which aligns visual representations with both semantics and actionsThe second stage is a causal diffusion transformer that predicts future visual latents together with the latent actions connecting them
• Representation Learningframework80%
  The paper introduces a semantic visual-action tokenizer that learns a shared representation for vision and actions, which is a form of representation learning.
  we introduce a semantic visual-action tokenizer, which aligns visual representations with both semantics and actionsLingBot-VA 2.0 places world states and latent actions in a single semantic latent space anchored to a language-aligned visual foundation model
• Transformer Architectureframework80%
  The paper uses a diffusion transformer (DiT) architecture, which is a transformer-based architecture.
  The second stage is a causal diffusion transformer that predicts future visual latents together with the latent actions connecting them
• Diffusion Modelsframework80%
  The paper uses diffusion models (flow matching) for video and action generation, and distills them into consistency models.
  The post-trained policy samples each chunk by integrating the flow-matching velocity field along the probability-flow (PF) ODEWe distill it into a few-step consistency model

Abstract

The advent of video-action models offers a promising path for robot control. Nevertheless, we argue that repurposing video generative models designed for digital content creation is inherently inadequate for physical environments. To bridge this gap, we present LingBot-VA 2.0, a video-action foundation model built from the ground up for embodiment. Four core design principles showcase its evolution from LingBot-VA. (1) Departing from traditional reconstruction-focused VAEs, we introduce a semantic visual-action tokenizer, which aligns visual representations with both semantics and actions, improving instruction following and action precision in subsequent policy learning. (2) Given the strictly causal nature of temporal dynamics, we adopt a causal pretraining paradigm, training from scratch to circumvent the catastrophic forgetting that frequently occurs when adapting bidirectional architectures. (3) To meet the demands of high-frequency inference, our model employs a sparse MoE backbone, expanding model capacity without compromising efficiency. (4) Real-time closed-loop control is realized through an enhanced asynchronous inference scheme, which predicts future latents in parallel with action execution while re-grounding each rollout on the latest observation via learned forward dynamics. Real-world deployment validates LingBot-VA 2.0 as a robust foundation model, as evidenced by its few-shot generalization across complex manipulation tasks.

Paper Context