The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs
2026-07-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors studied why vision-language models (VLMs), which understand images and text together, often fail at counting objects correctly. They found that these models usually do have the correct counting information inside, but fail to express it properly in their answers. By analyzing the models' internal activations, the authors showed that correcting these misaligned signals can improve counting without retraining. They also developed a method that spots counting mistakes during inference and fixes them, boosting accuracy significantly.
vision-language modelsobject countingactivationsnonlinear probesSVCCAcausal interventionself-correctioninference-timeerror detectionrepresentation alignment
Authors
Ahmed Oumar El-Shangiti, Abzal Nurgazy, Hilal AlQuabeh, Nikolai Rozanov, Kentaro Inui
Abstract
Despite strong performance on many multimodal tasks, vision-language models (VLMs) still struggle with basic object counting. We investigate whether this reflects missing internal knowledge or a gap between internal representations and verbalized outputs. Training simple probes on activations from four VLMs across five counting datasets reveals that nonlinear probes can reliably detect counting errors, suggesting that VLMs often encode the correct count even when they output the wrong answer. SVCCA analysis shows that probes trained on ground-truth counts and probes trained on model outputs occupy a partially shared activation subspace but read out along misaligned directions. We further validate our findings using a causal steering intervention, proving that strengthening the direction of count-identified probes does improve model counting performance. Motivated by this result, we propose a detector-guided self-correction method that selectively re-prompts the model only when an internal error detector predicts failure. This simple inference-time intervention improves counting accuracy by up to 15.6 absolute percentage points, without any parameter updates. Our results establish activation-based error probing as both a practical tool for improving VLM counting and a mechanistic lens on the gap between internal knowledge and model outputs.