When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics
2026-06-02 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors address the problem of high computation in vision-language models by improving how visual tokens (small parts of an image) are selected during processing. They found that using just attention scores makes the model focus only on similar areas, losing important details. Their method, called Structure-to-Semantics (STS), first picks tokens spread out across the image and then filters them based on relevance to the task. This two-step approach helps keep diverse and meaningful image information while reducing unnecessary data.
Vision-Language ModelsVisual Token PruningAttention ScoresFeature DiversitySpatial SamplingCross-AttentionSemantic RelevanceInference EfficiencyRepulsion-based Sampling
Authors
Jiahui Wang, Kai Zhang, Mai Han, Huanghe Zhang
Abstract
Vision-Language Models (VLMs) have demonstrated remarkable capabilities but suffer from significant computational overhead during inference. While visual token pruning offers a promising solution, existing methods predominantly rely on initial attention scores. This single-metric paradigm presents a critical flaw: high attention scores inherently collapse onto semantically similar regions, thereby severely reducing feature diversity and discarding vital contextual details. To address this, we introduce Structure-to-Semantics (STS), a novel two-stage visual token pruning framework that explicitly decouples the pruning process. The first stage employs a repulsion-based sampling mechanism to maximize spatial and structural diversity. The second stage leverages instruction-aware cross-attention to precisely filter out prompt-irrelevant tokens. This two-stage synergy constitutes the core of STS, first ensuring geometric coverage and then refining the retained tokens according to semantic relevance. Extensive evaluations demonstrate that STS mitigates the redundancy caused by attention-based selection, improving both structural diversity and fine-grained task alignment of the preserved visual tokens.