What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks
2026-07-11 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMultimedia
AI summaryⓘ
The authors found that short-answer visual question answering (VQA) tests mix up whether a model's answer is truly correct in meaning and whether it exactly matches the expected wording. By checking over 37,000 answers with a human-like judge, they saw many answers marked wrong just because their wording was different, not because they were wrong in meaning. This issue was worse for answers that pull directly from text or have multiple parts, and small changes in how the question is asked could flip results. The authors suggest that VQA scores should be checked with meaning-focused reviews to better understand a model’s real performance.
Visual Question Answering (VQA)Semantic correctnessSurface formAutomatic evaluationExtractive answersMulti-span answersFalse negativesBenchmark evaluationPrompt engineeringAnswer-type diagnostics
Authors
Guanhua Ye, Niu Jingbin, Yan Li, Meiyu Liang, Zhe Xue, Yingxia Shao, Yawen Li
Abstract
Short-answer VQA benchmarks conflate two distinct quantities: whether a model's answer is semantically correct, and whether that answer matches the surface form expected by the automatic evaluator. We study this conflation across six vision--language models and six benchmarks, using a human-validated semantic judge (97.6% precision) to audit over 37k official errors. A second text-only judge reproduces the same benchmark-level false-negative pattern, showing that the effect is not an artifact of a single audit model. On text-rich benchmarks, up to half of these errors are semantically acceptable answers penalized purely for surface-form mismatch. This instability is structured by answer type: extractive and multi-span answers are far more evaluator-sensitive than scalar answers. Benign prompt and context rewrites further destabilize official outcomes, flipping item-level correctness at substantial rates without changing the underlying task. A deterministic CPU-only contract repair confirms that the undercount is partially recoverable. These findings imply that official short-answer VQA scores should be accompanied by semantic audits and answer-type diagnostics to remain interpretable.