Natural Language Camera Movement Understanding
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how well current vision-language models understand camera movements described in natural language, like panning or rotating. They found that these models often mix up different types of movements and confuse the camera's motion with objects moving in the scene. To improve this, the authors created a clear system to categorize camera movements, a detailed test set with real and fake videos, and a large training dataset with special data to help models learn better. Their improved model did better than a strong competitor but still isn't as good as humans, showing more work is needed in this area.
vision-language modelscamera movementnatural language understandingvideo generationtranslationrotationobject movementbenchmark datasetdata augmentationfine-tuning
Authors
Yuwen Tan, Joey Huang, Jin Huang, Haoxiang Li, Boqing Gong
Abstract
Understanding camera movement in natural language is critical for training and evaluating video generation models, among other applications. However, we demonstrate that existing vision-language models (VLMs) fail this task in surprising ways, frequently confusing translation with rotation, left with right, and object movement with camera movement. To address these limitations, we establish natural language camera movement understanding as a standalone research task. We introduce a two-level cinematographic taxonomy and an extensive, atomic benchmark featuring both real and synthetic videos. Furthermore, we curate a large-scale, multi-source training set enhanced by targeted camera movement augmentation. Our fine-tuned VLM-8B outperforms Gemini 3.1 Pro by 10% and 11% on our benchmark's real and synthetic videos, respectively. Despite these gains, a significant gap remains relative to human performance, underscoring the need to promote and facilitate future research on natural language camera movement understanding.