Benchmarking Visual State Tracking in Multimodal Video Understanding
2026-06-02 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created a new test called VSTAT to see how well Multimodal Large Language Models (MLLMs) can follow changes and events throughout entire videos, instead of just looking at single pictures. They found that even top MLLMs do much worse than humans on this task, mostly because the models struggle to actually see and understand what is happening visually over time. The authors also checked if newer AI approaches could fix this problem, but those methods didn't improve performance much either. Overall, the study shows that current models have trouble tracking visual states across videos.
Multimodal Large Language ModelsVisual State TrackingVideo UnderstandingBenchmarkSynthetic VideosReal-world VideosContinuous PerceptionAgentic ApproachesThinking TracesVideo Agents
Authors
Sihyun Yu, Nanye Ma, Pinzhi Huang, Hyunseok Lee, Shusheng Yang, June Suk Choi, Ellis Brown, Oscar Michel, Boyang Zheng, Jinwoo Shin, Saining Xie
Abstract
Understanding a video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacity for visual state tracking is fundamental to video understanding, yet remains underexplored in current evaluations of Multimodal Large Language Models (MLLMs). We introduce Visual STAte Tracking benchmark (VSTAT), a video-based benchmark designed to diagnose visual state tracking in MLLMs. VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment, requiring continuous perception and integration of events across the entire video stream. Despite their strong performance on existing video benchmarks, we find that state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines. To analyze this gap, we compare MLLMs' thinking traces with the underlying video stream to understand why and when MLLMs fail on VSTAT. We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track. Finally, our preliminary evaluation suggests that recent agentic approaches, including MLLM-based video agents and coding agents, do not readily resolve these failures, still falling short on VSTAT.