EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies

Noosaga Placements

• Robot Manipulationsubfield95%
  The benchmark evaluates generalist manipulation policies on mobile pick-and-place, long-horizon multi-stage, and dexterous-and-precise tasks, directly targeting robot manipulation capabilities.
  EBench comprises 26 diverse and challenging manipulation tasksthree families: Mobile Pick-and-Place, Mobile Long-Horizon, and Table-Top Dexterous-and-Precise
• Learning-Based Manipulationframework90%
  The benchmark evaluates learning-based manipulation policies (VLAs) that are trained and evaluated on manipulation tasks in simulation.
  We evaluate state-of-the-art generalist manipulation models including π0, π0.5, XVLA, and InternVLA-A1All models are fine-tuned from pretrained checkpoints on the same EBench training data
• Roboticssubfield90%
  The paper evaluates vision-language-action models (VLAs) for generalist robotics, situating the benchmark within the broader robotics AI field.
  generalist mobile manipulation policiesVision–language–action models
• Deep Reinforcement Learningframework60%
  The paper discusses pretraining and fine-tuning of VLA models, which can involve deep reinforcement learning principles, but the benchmark itself does not directly use RL methods.
  post-training dataset200K gradient steps, batch size 128, AdamW optimizer
• Robot Learningsubfield80%
  The benchmark is used to evaluate robot learning models (VLAs) post-trained on demonstration data, and analyzes generalization and pretraining effects.
  post-training dataset contains 91.4 hours demonstrationspretraining ablation across EBench, LIBERO, and RoboTwin 2.0
• Mobile Roboticssubfield70%
  Many tasks involve mobile manipulation with a dual-arm robot on a mobile base, requiring base motion coordination.
  mobile pick-and-place tasksmobile manipulation is hard to teleoperate since a single operator has to coordinate base motion and arm motion

Abstract

We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions. We evaluate state-of-the-art generalist manipulation models including $π_0$, $π_{0.5}$, XVLA, and InternVLA-A1, and reveal that models with near success rates exhibit strikingly different capability profiles: $π_{0.5}$ achieves the highest test success rate and the best train--test retention, whereas InternVLA-A1 dominates mobile manipulation but collapses on dexterous tasks, and XVLA exhibits strengths on a disjoint set of atomic skills compared to other policies. Beyond capability profiling, EBench analyzes the generalization ability from 4 representative perspectives, identifying the impact of different distribution shift factors. The results reveal strengths and weaknesses of models behind an overall score. We hope this benchmark offers a broad set of diagnostic signals to guide iteration on generalist manipulation models.

Paper Context