Self-Prophetic Decoding to Unlock Visual Search in LVLMs
2026-05-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors study Large Vision-Language Models (LVLMs) and their challenges in visual search tasks, especially when reasoning over multiple steps. They find that combining the model’s capabilities before and after extra training helps reduce problems like capability decline and confusion in longer reasoning. They propose a new method called SeProD that uses a smart sampling approach, letting the first model suggest possible outputs and the second model decide which to keep, improving reasoning without extra training or slowdown. Their experiments show SeProD makes visual search and question-answering tasks better across several datasets.
Large Vision-Language ModelsVisual SearchMultimodal ReasoningSelf-RegulationPost-TrainingProphetic SamplingMulti-Step ReasoningVision Question AnsweringDecoding MethodsPlug-and-Play Framework
Authors
Zhendong He, Qiyuan Dai, Guanbin Li, Liang Lin, Sibei Yang
Abstract
Large Vision-Language Models (LVLMs) are rapidly evolving toward true multimodal reasoning, with visual search representing a concrete instantiation of the thinking-with-images paradigm. However, LVLM visual search faces two key challenges: incompatibility among intrinsic capabilities after post-training, and interference in long multi-step reasoning contexts. To address these, we identify two novel insights. First, self-regulation between pre- and post-training LVLMs leverages the intrinsic single-step capabilities of the pre-training model to mitigate capability deterioration and long-context interference. Second, probability-based prophetic sampling, replacing naive prompting, provides a probabilistic interface where the pre-training model acts as a prophet and the post-training model selectively accepts prophetic tokens under its output distribution, preserving coherent multi-step reasoning. Building on these insights, we introduce SeProD, a self-prophetic decoding framework that leverages intrinsic single-step capabilities to enable coherent multi-step reasoning in a training-free, plug-and-play manner. Experiments show that SeProD consistently improves multiple visual-search LVLMs across all 12 splits of 4 visual search benchmarks, as well as across general VQA benchmarks, without added computational overhead, thanks to its parallel prophetic acceptance mechanism.