EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies
2026-06-16 • Robotics
Robotics
AI summaryⓘ
The authors created EBench, a test suite with 26 different tasks to better understand how well general robot manipulation programs perform beyond just success rates. They evaluated several top models and found that each has unique strengths and weaknesses in different skills and types of generalization. For example, one model is good at keeping its performance from training to testing, while another excels in mobile tasks but struggles with precise movements. EBench helps show these differences clearly, providing insights to improve future robot manipulation systems.
mobile manipulationgeneralist policiesbenchmarkcapability profilinggeneralizationdistribution shiftsuccess raterobotics tasksperformance evaluation
Authors
Ning Gao, Jinliang Zheng, Xing Gao, Haoxiang Ma, Hanqing Wang, Yukai Wang, Jiantong Chen, Zanxin Chen, Shujie Zhang, Mingda Jia, Xuekun Jiang, Zihou Zhu, Xinyu Li, Shuai Wang, Hao Li, Wenzhe Cai, Yuqiang Yang, Xudong Xu, Zhaoyang Lyu, Yao Mu, Tai Wang, Jiangmiao Pang, Jia Zeng, Weinan Zhang, Chunhua Shen
Abstract
We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions. We evaluate state-of-the-art generalist manipulation models including $π_0$, $π_{0.5}$, XVLA, and InternVLA-A1, and reveal that models with near success rates exhibit strikingly different capability profiles: $π_{0.5}$ achieves the highest test success rate and the best train--test retention, whereas InternVLA-A1 dominates mobile manipulation but collapses on dexterous tasks, and XVLA exhibits strengths on a disjoint set of atomic skills compared to other policies. Beyond capability profiling, EBench analyzes the generalization ability from 4 representative perspectives, identifying the impact of different distribution shift factors. The results reveal strengths and weaknesses of models behind an overall score. We hope this benchmark offers a broad set of diagnostic signals to guide iteration on generalist manipulation models.