ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation

Noosaga Placements

• Diffusion Modelsframework90%
  The paper explicitly uses diffusion models for motion generation, employing a DDPM framework with a denoising process.
  we introduce ARDY, an Auto-Regressive Diffusion modelwe train the denoiser using the DDPM framework [Ho et al. 2020]
• Computational Imagingsubfield90%
  The paper focuses on generating 3D human motion for animation and simulation, which is a core topic in computer graphics. The primary arXiv category is cs.GR (Graphics).
  Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics.CCS Concepts: • Computing methodologies → Motion processing.
• Transformer Architectureframework90%
  The denoiser is based on transformer architecture, using transformer encoder layers with causal attention.
  We propose a two-stage autoregressive transformer denoiserBoth the root and body transformer in our two-stage denoiser employ the same transformer encoder architecture.
• Deep Learningsubfield90%
  The method heavily relies on deep learning architectures: autoregressive transformers, diffusion models, and a learned tokenizer. The paper explicitly discusses transformer and diffusion model design.
  ARDY utilizes an autoregressive transformer denoiser for interactive motion generationWe propose a two-stage autoregressive transformer denoiser that features variable history contextthe denoising process at step k can be written as: ...
• Representation Learningframework80%
  The paper uses a learned tokenizer to create a compact latent representation of body motion, which is a form of representation learning.
  a motion tokenizer first learns a compact latent representation of body motionthe latent body representation is more compact than explicit representations
• Computer Visionsubfield60%
  The paper uses text-to-motion generation, which relates to vision-language tasks, and evaluates on HumanML3D, a common computer vision benchmark. However, the core focus is on motion generation for graphics, not visual perception.
  Extensive evaluations on the HumanML3D benchmarktext-conditioned motion generation
• Roboticssubfield50%
  The paper mentions applications in humanoid robotics and uses kinematic constraints relevant to robot control, but the method itself is a kinematic motion generator, not a robotics system.
  Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics.recent work in real-world humanoid robot control ... relies heavily on high-quality human motions for supervision

Abstract

Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations on the HumanML3D benchmark and the large-scale, high-fidelity Bones Rigplay dataset demonstrate ARDY's high motion quality and constraint adherence, validating the efficacy of our key architectural decisions. Finally, we demonstrate the method's practical versatility through an interactive demo featuring dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard. Supplementary video results, code, and model releases can be found at https://research.nvidia.com/labs/sil/projects/ardy/.

Paper Context