AI summaryⓘ
The authors studied why vision-language models (VLMs) struggle with counting objects accurately, even though other tasks are easier for them. They found that combining these models with object detection models (ODMs) like YOLO, which are good at spotting and counting items, helps reduce mistakes. Their method, called GroundCount, uses clear spatial information from ODMs to improve counting accuracy and speed. They also discovered that how the information is added matters: some techniques work better for stronger models, while others can cause problems. Overall, their work shows that counting errors come from difficulty linking object positions with their meanings inside the models, not from just the model design itself.
Vision Language ModelsObject Detection ModelsCounting TasksSpatial GroundingHallucinationsPrompt-based AugmentationPositional EncodingFeature FusionCross-Attention MechanismsIterative Reflection
Authors
Boyuan Chen, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique
Abstract
Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-art reasoning-capable VLMs. Conversely, CNN-based object detection models (ODMs) such as YOLO excel at spatial localization and instance counting with minimal computational overhead. We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations. In the best case, our prompt-based augmentation strategy achieves 81.3% counting accuracy on the best-performing model (Ovis2.5-2B) - a 6.6pp improvement - while reducing inference time by 22% through elimination of hallucination-driven reasoning loops for stronger models. We conduct comprehensive ablation studies demonstrating that positional encoding is a critical component, being beneficial for stronger models but detrimental for weaker ones. Confidence scores, by contrast, introduce noise for most architectures and their removal improves performance in four of five evaluated models. We further evaluate feature-level fusion architectures, finding that explicit symbolic grounding via structured prompts outperforms implicit feature fusion despite sophisticated cross-attention mechanisms. Our approach yields consistent improvements across four of five evaluated VLM architectures (6.2--7.5pp), with one architecture exhibiting degraded performance due to incompatibility between its iterative reflection mechanisms and structured prompts. These results suggest that counting failures stem from fundamental spatial-semantic integration limitations rather than architecture-specific deficiencies, while highlighting the importance of architectural compatibility in augmentation strategies.