WorldSample: Closed-loop Real-robot RL with World Modelling

Noosaga Placements

• Robot Learningsubfield95%
  The paper focuses on improving real-robot reinforcement learning through a world-model-based data augmentation framework, which is a core topic in robot learning.
  WorldSample, a physically grounded data augmentation framework for real-robot RLWorldSample improves policy success rate by 28% while reducing training steps by 59% compared with baselines
• Model-Based Reinforcement Learningframework95%
  WorldSample uses a world model to generate synthetic transitions, which is a core component of model-based reinforcement learning.
  WorldSample generates high-fidelity synthetic transitions through a post-trained world modelWorldSample, a physically grounded data augmentation framework for real-robot RL that closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement
• Reinforcement Learningsubfield90%
  The paper proposes a method to improve reinforcement learning efficiency for real robots, using RL as the core learning paradigm.
  Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL)WorldSample, a physically grounded data augmentation framework for real-robot RL
• Reinforcement Learningframework90%
  The paper uses reinforcement learning as the core learning paradigm for policy improvement.
  Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL)WorldSample, a physically grounded data augmentation framework for real-robot RL
• Robot Manipulationsubfield85%
  The method is evaluated on robot manipulation tasks including contact-rich and precise tasks.
  Experiments on robot manipulation tasks involving contact-rich and precise tasksPushing, Insertion, Sorting, Pick & Place, and Assemble
• Model-Free Reinforcement Learningframework70%
  The paper compares WorldSample against model-free RL baselines (HIL-SERL) and discusses the limitations of model-free methods.
  Compared with HIL-SERL, WorldSample significantly improves the success rate on all five tasksHIL-SERL is our real-robot RL baseline
• Machine Learningsubfield60%
  The paper uses machine learning techniques such as world models and data augmentation, but the primary focus is on robotics and RL.
  WorldSample generates high-fidelity synthetic transitions through a post-trained world modelPolicy-Paced Learning (PPL) to regulate the training process through sample selection and scheduling
• Imitation Learningframework60%
  The paper discusses imitation learning as a baseline and compares against it, but does not use it as the primary method.
  Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL)VLAW ... we adapt VLAW as an imitation-learning baseline

Abstract

Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL) by allowing robots to improve through trial-and-error interaction beyond the states observed in demonstrations. However, deploying RL on real robots remains constrained by high interaction costs, since each physical rollout is costly and reflects only one realized action-outcome path. To address this challenge, we propose WorldSample, a physically grounded data augmentation framework for real-robot RL that closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement. Grounded on real rollouts, WorldSample generates high-fidelity synthetic transitions through a post-trained world model, which greatly lowers the visual hallucination. Specifically, rather than simply using these transitions as real-world experience, WorldSample introduces Policy-Paced Learning (PPL) to regulate the training process through sample selection and scheduling, balancing useful augmentation against value overestimation and mitigating the hallucination-induced noise. Experiments on robot manipulation tasks involving contact-rich and precise tasks show that WorldSample improves policy success rate by 28% while reducing training steps by 59% compared with baselines. Furthermore, WorldSample improves world model visual fidelity by 19.4dB in PSNR and 0.47 in SSIM over demonstration-only post-training, validating the effectiveness of the real-synthetic loop for both policy and world model performance.

Paper Context