RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

Noosaga Placements

• Reinforcement Learningsubfield95%
  The paper proposes an offline reinforcement learning framework (NF-IQL) that uses advantage-weighted likelihood optimization, building on IQL and addressing distribution shift in offline RL.
  Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction.NF-IQL post-training, which directly increases the exact likelihood of high-advantage actions from the offline dataset.
• Deep Reinforcement Learningframework95%
  The paper builds on Implicit Q-Learning (IQL), an offline RL method, and extends it with advantage-weighted likelihood optimization, which is a deep RL technique.
  RoMAN-Flow builds upon Implicit Q-Learning (IQL) (Kostrikov, Nair, and Levine 2021)NF-IQL post-training, which directly increases the exact likelihood of high-advantage actions
• Robot Manipulationsubfield90%
  The framework is specifically designed for robotic manipulation, evaluated on manipulation benchmarks (MetaWorld, LIBERO, RoboMimic) and real-world manipulation tasks with a Franka arm and XHand.
  RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulationExperiments across multiple simulated manipulation benchmarks and real-world robotic platforms
• Learning-Based Roboticsframework90%
  The paper proposes a learning-based approach for robot policy learning, using neural networks and offline RL, which aligns with learning-based robotics.
  RoMAN-Flow, an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulationExperiments across multiple simulated manipulation benchmarks and real-world robotic platforms
• Data-Driven and Learning-Based Manipulationframework85%
  The method is a data-driven, learning-based approach for robotic manipulation, using offline datasets and neural policies.
  Offline reinforcement learning improves robotic policies using previously collected dataExperiments across multiple simulated manipulation benchmarks and real-world robotic platforms
• Robot Learningsubfield80%
  The paper focuses on learning robot policies from offline data, combining imitation learning and offline RL, which falls under robot learning.
  We consider imitation learning followed by offline reinforcement learning post-training.RoMAN-Flow is trained in three stages: imitation learning, NF-IQL post-training, and one-step distillation.
• Generative Deep Learningframework80%
  The paper uses autoregressive normalizing flows, a type of generative deep learning model, for action generation.
  AR-NFs offer both expressive action modeling and exact likelihood evaluationwe adopt SimFlow as the backbone of our conditional AR-NF model
• Deep Learningsubfield70%
  The method heavily relies on deep learning architectures, including transformer-based AR-NFs, vision-language encoders, and distillation techniques.
  Recent transformer-based autoregressive normalizing flows (AR-NFs) have demonstrated strong generative modeling capabilitywe employ a pretrained vision-language model as the multimodal encoder
• Deep Generative Modelsframework75%
  The paper uses normalizing flows, which are deep generative models, to model action distributions.
  AR-NFs offer both expressive action modeling and exact likelihood evaluationNormalizing flows instead provide exact density evaluation through invertible transformations
• Machine Learningsubfield60%
  The work involves generative modeling with normalizing flows and likelihood-based optimization, which are core machine learning topics.
  AR-NFs offer both expressive action modeling and exact likelihood evaluationNormalizing flows instead provide exact density evaluation through invertible transformations
• Reinforcement Learningframework70%
  The paper applies offline reinforcement learning to robot learning, which is a key topic in robot learning.
  Offline reinforcement learning improves robotic policies using previously collected dataNF-IQL post-training, which directly increases the exact likelihood of high-advantage actions

Abstract

Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.

Paper Context