VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimization
2026-06-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how video generation models (VGMs) can better solve reasoning tasks involving video by using vision-language models (VLMs) not as direct solvers, but as teachers that guide the VGMs. Their method has VLMs extract task rules and create rewards that help optimize the VGM's performance during testing, improving logical accuracy. This approach was tested on video reasoning benchmarks and showed a significant improvement over previous methods that tried to use VLMs to produce or refine instructions directly. Essentially, the authors propose shifting VLMs' role to guiding VGMs in real-time, which helps handle complex task rules better.
Video Generation Models (VGMs)Vision-Language Models (VLMs)Video ReasoningSpatiotemporal ReasoningTest-time OptimizationLoRA ModuleReward-based LearningSymbolic Reasoning BenchmarksVision-Language IntegrationDifferentiable Rewards
Authors
Junhao Cheng, Liang Hou, Tianxiong Zhong, Xin Tao, Pengfei Wan, Kun Gai, Jing Liao
Abstract
The recent "Reasoning with Video" paradigm utilizes Video Generation Models (VGMs) to generate temporally coherent visual trajectories to complete reasoning tasks. Although state-of-the-art VGMs excel at visual quality, they often struggle to understand and follow task-specific rules, leading to logical failures across diverse reasoning scenarios. Existing efforts try to utilize Vision-Language Models (VLMs) as problem pre-solvers to produce or refine textual guidance for the VGM. However, textual descriptions fail to capture intricate spatiotemporal details, and VGMs often struggle to faithfully execute fine-grained or long-tail instructions even with a valid plan. While VLMs struggle as solvers, they possess strong perception capabilities to evaluate process-constraint satisfaction and final-goal achievement. Leveraging this strength, we introduce a paradigm shift that transitions the role of VLMs to "teachers". Specifically, a VLM teacher extracts task-specific rules to formulate differentiable rewards, guiding a VGM Reasoner via test-time online optimization of a lightweight LoRA module. This strategy enables adaptive test-time optimization and extends the reasoning capabilities beyond the VGM's intrinsic boundaries. Evaluations on symbolic (VBVR-Bench) and general-purpose (RULER-Bench) video reasoning benchmarks show that the proposed method yields a 16.7-point average performance gain, outperforming the VLM-as-Solver paradigm (+0.4 points) and Best-of-N scaling (+2.2 points) by a large margin at comparable test-time cost. These findings reveal that integrating VLMs as test-time teachers offers a promising paradigm for achieving generalizable video reasoning. Project Page: https://VLM-as-Teacher.github.io/