GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
2026-06-03 • Robotics
Robotics
AI summaryⓘ
The authors created GRAIL, a fully virtual system that generates realistic robot interaction data without needing physical setups or real-world recordings. They use detailed 3D scenes and video models to simulate how a humanoid robot can pick up objects, move, and navigate terrain. By accurately tracking human-object interactions in these virtual scenes, they train robots to perform tasks like object manipulation and stair climbing. The trained robots were tested in the real world and succeeded most of the time, showing the approach can prepare robots for complex actions without extensive physical trials.
Humanoid robotLoco-manipulation3D simulationVideo foundation modelsHuman-object interactionSim-to-real transferRobot teleoperationMotion trackingTask-general policyTerrain traversal
Authors
Tianyi Xie, Haotian Zhang, Jinhyung Park, Zi Wang, Bowen Wen, Jiefeng Li, Xueting Li, Qingwei Ben, Haoyang Weng, Yufei Ye, David Minor, Tingwu Wang, Chenfanfu Jiang, Sanja Fidler, Jan Kautz, Linxi Fan, Yuke Zhu, Zhengyi Luo, Umar Iqbal, Ye Yuan
Abstract
Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation. We present GRAIL, a digital generation pipeline that remains fully virtual until deployment: it composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot. Rather than reconstructing unconstrained in-the-wild videos, GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction. This privileged setup better conditions 4D recovery, allowing model-based object tracking, human motion estimation, and interaction-aware optimization to reconstruct metric 4D human-object interaction (HOI) trajectories with reduced depth ambiguity and morphology mismatch. We retarget the recovered motions to a humanoid robot and train complementary task-general trackers: an object-aware latent adaptor for manipulation and a scene-aware tracker for terrain traversal. GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting, and terrain traversal. Using only GRAIL-generated data, we train egocentric visual policies through a sim-to-real pipeline and deploy them on a Unitree G1 humanoid, achieving 84\% real-world success on diverse object pick-up and 90\% success on stair-climbing.