Robot-Factored World Models via Robot Rendering
2026-07-24 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors address how robots predict future video frames based on actions by separating the robot's own movement and appearance from the environment's reaction. Instead of having the model learn how actions turn into movements, they use the robot’s controller to generate a clear motion path and render the robot's shape and position explicitly. This approach helps the model focus on how objects in the scene respond to the robot, improving prediction quality and allowing it to work with different robots or even human hand motions. They also use depth information to help understand how objects touch or hide each other. Their experiments show their method works better than models that directly use action commands.
video world modelsaction-conditioned predictionrobot kinematicsrobot controllerURDFdepth perceptionrobot renderingmanipulation videosgeneralizationrobot embodiment
Authors
Byungjun Kim, Taeksoo Kim, Hyunsoo Cha, Hanbyul Joo
Abstract
Action-conditioned video world models predict future observations from an initial observation and an action signal. In robotics, actions influence future observations through two distinct processes: they are first realized into robot motion by the robot body and controller, and the scene then responds through contact and object motion. Conditioning directly on action commands asks the world model to learn the realization process itself, while conditioning on logged future states leaks the interaction outcomes it is meant to predict. We propose robot-factored world models, which move two robot-specific factors outside the world model. First, action realization: each command is rolled through the robot's own controller and kinematics into a deployment-available nominal trajectory, a middle signal that avoids both action-realization learning and future-state leakage. Second, robot rendering: this nominal trajectory is rendered through the robot URDF, factoring the robot's geometry, kinematics, and appearance out of the model and into explicit rendered robot geometry. To resolve depth ambiguity, we pair end-effector depth with scene depth, giving geometric cues for contact and occlusion beyond image-plane overlap. Together, camera-aware static RGB/depth context and rendered robot geometry form a shared visual world-model interface that stays consistent across viewpoints and robot embodiments, so the model sees the action only as visible robot geometry and learns how objects respond to it. Our experiments show that the rendered interface outperforms vector-conditioned baselines and generalizes to unseen robot embodiments at inference. We further demonstrate that our model generates robot manipulation videos from human demonstrations by retargeting and rendering the hand motion as robot geometry.