SimWAM: A Simple World Action Model for End-to-End Autonomous Driving
2026-08-07 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed SimWAM, a method to improve autonomous driving by learning from videos during training but not needing to generate future frames when driving. They combined a video model and an action model so that after training, only the simpler action model is used for predicting driving paths. This approach makes driving predictions faster and still accurate, outperforming previous similar methods. The authors also used reinforcement learning to make the driving better than just copying data. SimWAM works well on different driving tests and their code is publicly available.
autonomous drivingvideo generationaction predictionend-to-end learningflow matchingreinforcement learningtrajectory predictionzero-shot transferNAVSIMnuScenes
Authors
Zongchuang Zhao, Xin Zhou, Tianyang Xu, Zhengyang Sun, Kaixuan Zhou, Honglin Li, Dingkang Liang, Xiang Bai
Abstract
World-Action Models (WAMs) improve end-to-end autonomous driving by transferring video dynamics priors to action prediction, but existing methods require costly future generation at inference. We present SimWAM, a simple yet effective WAM that uses video generation purely as a training signal. It co-trains a pretrained video expert and a lightweight action expert with joint flow matching. An isolated attention mask keeps action prediction independent of future frames, allowing the video branch to be discarded after training and leaving a self-contained planner that directly predicts trajectories. Since the two experts share no parameters and interact only through a unified attention interface, the video backbone could be replaced and the action expert scaled independently without modifying the learning objective or inference pipeline. We further apply reinforcement learning to optimize a compositional driving reward beyond trajectory imitation. Our SimWAM achieves $91.5$ PDMS on NAVSIM, surpasses state-of-the-art WAM-based planners with substantially lower latency, and transfers zero-shot to nuScenes. These results position SimWAM as a simple yet solid baseline that could readily benefit from advances in video generation for efficient autonomous driving. The code and model weights are available at https://github.com/H-EmbodVis/SimWAM/