DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation
2026-08-13 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionRobotics
AI summaryⓘ
The authors present DreamX-Phi 1.0, a video model that predicts what a robot will see after performing a series of actions guided by language instructions. To make sure the robot's movements are accurately represented, they use special geometric encodings that keep track of each robotic arm's motion. They also add extra parts to understand depth and keep objects consistent during manipulation. Finally, they make the model smaller for faster use, achieving top results in a robotics challenge.
robotic manipulationvideo world modelSE(3) transformationsgeometric encodingdepth estimationobject segmentationSAM3 masksV-JEPAdistillationaction-conditioned prediction
Authors
DreamX Team, Rui Chen, Xiangxiang Chu, Geng Li, Jifan Li, Qingfeng Shi, Datao Tang, Jing Tang, Jun Wang, Pengfei Zhang
Abstract
We present \textbf{DreamX-Phi 1.0}, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm's commanded path, we inject per-arm $\mathrm{SE}(3)$ transformations into attention via \textbf{PRoPE-style geometric encoding}, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweight \textbf{depth branch} for scene-level geometry and use \textbf{SAM3 masks} with a frozen \textbf{V-JEPA teacher} to maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student via distribution-matching distillation for efficient deployment. At the time of writing, \model{} achieves first place on Track~1 and second place on Track~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.