ReToken: One Token to Improve Vision-Language Models for Visual Retrieval
2026-07-30 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors address a problem where vision-language models struggle when dealing with lots of visual information because it's hard to process everything at once. They introduce ReToken, a special small piece of information trained to pick out only the important visual parts from a big collection, making it easier for the model to focus. Even though ReToken was trained on a small dataset, it improves performance on several tests involving images and long videos, and it works efficiently on common hardware. This approach helps models handle long visual contexts better without needing too much computing power.
vision-language modelsvisual tokenskey-value cacheimage question answeringlong visual contexttoken retrievalvideo benchmarkszero-shot transferGPU memory constraintsembedding
Authors
Yao Xiao, Reuben Tan, Zhen Zhu, Yuqun Wu, Jianfeng Gao, Derek Hoiem
Abstract
Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints. We present ReToken, a single learnable embedding trained as an explicit retrieval target that selects a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. Trained on only a small image-QA dataset, ReToken yields consistent gains across image and video benchmarks: on Visual Haystacks it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative), and on LVBench it transfers zero-shot to long video for an 8.0-point gain with Qwen3VL-8B. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken