FlowWAM: Optical Flow as a Unified Action Representation for World Action Models
2026-07-14 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors introduce FlowWAM, a new approach that uses optical flow videos to represent actions for models that predict and control video sequences. Unlike previous methods, their approach captures detailed motion information in a way compatible with pretrained video generators, improving predictions and world modeling. They show that FlowWAM works well even without labeled action data, using large unlabeled video datasets for training. Their experiments demonstrate better performance on manipulation tasks and video-based world modeling compared to existing methods.
World Action Modelsoptical flowvideo generationaction representationdiffusion modelpolicy modeworld modelingunlabeled video datatrajectory accuracy
Authors
Yixiang Chen, Peiyan Li, Yuan Xu, Qisen Ma, Jiabing Yang, Kai Wang, Jianhua Yang, Dong An, He Guan, Gaoteng Liu, Jianlou Si, Jun Huang, Jing Liu, Nianfeng Liu, Yan Huang, Liang Wang
Abstract
World Action Models (WAMs) are able to leverage pretrained video generators for both world modeling and action prediction. However, directly leveraging such video generators for control raises a new challenge: how to represent actions in a suitable form that aligns with pretrained video generators while carrying enough motion cues for accurate control. Existing numerical actions fail to satisfy the former, and prior visual action representations overlook the temporal motion structure across frames. We address this issue with FlowWAM, a dual-stream diffusion framework that adopts optical flow as a unified, video-native action representation. Flow videos share the same format as RGB videos and encode rich per-pixel displacement. By jointly modeling them within a shared pretrained video generator, FlowWAM can naturally implement two modes of WAMs. In policy mode, FlowWAM generates flow for action prediction, while in world-model mode, it uses target flow sequences to guide future video generation. Moreover, since flow can be easily extracted from raw videos without action labels, FlowWAM can leverage large-scale action-unlabeled video datasets for pretraining. We empirically find that our flow-based action representation delivers gains across both modes. On RoboTwin manipulation, FlowWAM raises the success rate to 92.94% on the Clean setting and 92.14% on Random, outperforming both VLA and WAM baselines. On WorldArena world modeling, it achieves the best overall EWMScore (63.71) with an 18.4% relative improvement in trajectory accuracy. More results can be found on our project website: https://flow-wam.github.io .