Visually-Guided Policy Optimization for Multimodal Reasoning
2026-04-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors looked at vision-language models and found that these models often don't pay enough attention to the visuals while reasoning, especially as they process information over time. To fix this, they created a method called Visually-Guided Policy Optimization (VGPO) that helps the model focus more on important visual parts throughout its reasoning steps. VGPO uses a strategy to boost visual cues and reduce forgetting of visual information. Their experiments show that this approach improves the model's attention to visuals and makes it better at tasks that require understanding both pictures and text, like math problems involving images.
Reinforcement LearningVision-Language ModelsVisual AttentionPolicy OptimizationVisual ForgettingMultimodal ReasoningVisual ActivationAdvantage Re-weighting
Authors
Zengbin Wang, Feng Xiong, Liang Lin, Xuecai Hu, Yong Wang, Yanlin Wang, Man Zhang, Xiangxiang Chu
Abstract
Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning ability of vision-language models (VLMs). However, the inherent text-dominated nature of VLMs often leads to insufficient visual faithfulness, characterized by sparse attention activation to visual tokens. More importantly, our empirical analysis reveals that temporal visual forgetting along reasoning steps exacerbates this deficiency. To bridge this gap, we propose Visually-Guided Policy Optimization (VGPO), a novel framework to reinforce visual focus during policy optimization. Specifically, VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues, while progressively elevating visual expectations in later steps to counteract visual forgetting. Building on this mechanism, we implement a dual-grained advantage re-weighting strategy: the intra-trajectory level highlights tokens exhibiting relatively high visual activation, while the inter-trajectory level prioritizes trajectories demonstrating superior visual accumulation. Extensive experiments demonstrate that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks.