$τ_0$-WM: A Unified Video-Action World Model for Robotic Manipulation
2026-05-31 • Robotics
Robotics
AI summaryⓘ
The authors developed a model called τ₀-World Model that helps robots plan and evaluate their actions by predicting future video outcomes. This model learns from many hours of robot and human videos, combining what it sees, instructions, and robot states to guess what will happen next. It can simulate different possible actions and choose the best one by comparing predicted outcomes. Their approach improves robot performance on complex tasks better than other methods.
robotic manipulationvideo predictionpolicy learningaction evaluationvideo diffusion modelmulti-view observationsegocentric videosteleoperationtask progress prediction
Authors
Pengfei Zhou, Shengcong Chen, Di Chen, Jiaxu Wang, Rongjun Jin, Bingwen Zhu, Yike Pan, Songen Gu, Kuanning Wang, Shufeng Nan, Xingyu Qiu, Chenhao Qiu, Pu Yang, Yunuo Cai, Jianxiong Gao, Yifan Li, Yanwei Fu, Xiangyu Yue, Zhi Chen, Jianlan Luo
Abstract
Robotic manipulation requires models that generate executable actions while anticipating and evaluating their future consequences before physical execution. We present $τ_0$-World Model ($τ_0$-WM), a unified video-action world model that integrates policy learning, video prediction, and action evaluation within a single future-predictive framework. Built on a shared video diffusion backbone, $τ_0$-WM provides two complementary interfaces. First, a video action model jointly predicts future visual latents and continuous action chunks from multi-view observations, language instructions, and robot state. Second, an action-conditioned video simulator rolls out candidate action chunks into multi-view futures and predicts dense task-progress scores. The model is trained on approximately $27{,}300$ hours of real-robot teleoperation, UMI-style interaction, egocentric human videos, and rollout or failure trajectories using modality-specific supervision masks. At inference time, $τ_0$-WM uses test-time computation to sample action candidates, rank them with re-denoising consistency, and invoke simulator-based rectification for low-quality candidates. On challenging long-horizon and fine-grained robotic manipulation tasks, $τ_0$-WM shows superior performance over other relevant baselines.