Continual Robot Policy Learning via Variational Neural Dynamics

Noosaga Placements

• Robot Learningsubfield95%
  The paper presents a method for learning robot policies from real-world interaction data, using a learned dynamics model and differentiable simulation for policy improvement. This directly falls under robot learning.
  We propose a continual learning framework that uses real-world experience to improve robot policies under hidden and recurring dynamics.Policy learning is performed via differentiable simulation using diverse learned dynamics sampled from the latent model.
• Model-Based Reinforcement Learningframework90%
  The paper explicitly uses a learned dynamics model for policy optimization via differentiable simulation, which is a form of model-based reinforcement learning. The policy is updated by backpropagating through the learned dynamics.
  Policy learning is performed via differentiable simulation using diverse learned dynamics sampled from the latent model.Following differentiable policy learning, we maximize maxθ Ez∼p(z), s0∼ρ0 [Σ γ^t r(st, at, st+1)].
• Robot Controlsubfield85%
  The method produces a control policy for quadrotor trajectory tracking and other tasks, and is compared against classical adaptive control (L1-MPC) and other control baselines. The policy is deployed at 50 Hz for real-time control.
  On real quadrotor trajectory tracking under changing wind, the policy recovers from recurring disturbances in roughly 1s.The policy is then conditioned on both the current task observation and the inferred latent: at = πθ(ot, zt).
• Learning-Based Robot Controlframework85%
  The paper proposes a learning-based approach to robot control, where the policy is a neural network conditioned on inferred latent dynamics. This directly falls under learning-based robot control.
  Our method learns a condition-aware dynamics model from real state-action trajectories... and this estimate conditions both the residual model and the policy.The policy is then conditioned on both the current task observation and the inferred latent: at = πθ(ot, zt).
• Robot Learningframework80%
  The paper is fundamentally about a robot learning from its own experience to improve its policy, which is the core of the Robot Learning framework.
  We propose a continual learning framework that uses real-world experience to improve robot policies under hidden and recurring dynamics.The framework combines an analytical physics prior with latent-conditioned neural residual dynamics.
• Reinforcement Learningsubfield70%
  The policy learning uses a reward signal and backpropagation through time (BPTT) over differentiable rollouts, which is a form of model-based reinforcement learning. The paper also compares against model-free RL methods.
  Following differentiable policy learning, we maximize maxθ Ez∼p(z), s0∼ρ0 [Σ γ^t r(st, at, st+1)].Model-free RL methods support online updates in principle but require sample budgets unaffordable on physical robots.
• Probabilistic Robot Learningframework75%
  The paper uses a variational approach with a probabilistic latent space and MMD regularization, which aligns with probabilistic robot learning. The dynamics model is learned with uncertainty via the latent variable.
  We introduce a Variational Neural Dynamics model that learns latent-conditioned residual dynamics from state-action trajectories.We align the aggregated encoder distribution over N context windows, qϕ(z) = 1/N Σ δ(z - Eϕ(Hn,0:C)) with the sampling prior p(z) = N(0, I) using maximum mean discrepancy (MMD).
• Machine Learningsubfield60%
  The paper uses variational inference concepts (MMD, latent variable models) and a recurrent encoder, which are machine learning techniques. However, the primary focus is on robotics applications.
  We introduce a Variational Neural Dynamics model that learns latent-conditioned residual dynamics from state-action trajectories.We align the aggregated encoder distribution with the sampling prior using maximum mean discrepancy (MMD).

Abstract

Robots deployed in the real world rarely operate under a single fixed dynamics model: wind changes, payloads vary, batteries drain, contacts shift, and hardware wears. Yet most learning-based controllers are trained once and deployed as if learning were complete. This prevents the robot from using deployment experience to further improve task performance. In this work, we propose a continual learning framework that uses real-world experience to improve robot policies under hidden and recurring dynamics. Our method learns a condition-aware dynamics model from real state-action trajectories by combining an analytical physics prior with a neural residual for unmodeled effects. A recurrent encoder infers the current hidden condition from recent interaction, and this estimate conditions both the residual model and the policy. Policy learning is performed via differentiable simulation using diverse learned dynamics sampled from the latent model. At deployment, these sampled conditions are replaced by conditions inferred online from recent real interaction, allowing the policy to recover recurring dynamics by recognition rather than residual re-fitting. Through extensive simulation studies and real-world experiments, we demonstrate that the framework improves policy performance under diverse unobserved disturbances. On real quadrotor trajectory tracking under changing wind, the policy recovers from recurring disturbances in roughly 1s, about 5x faster than online residual re-fitting. It also reduces large-disturbance hover and tracking errors by 65.7% and 53.3% over the state-of-the-art online adaptation approaches

Paper Context