SABRE: Scalable and Automated Benchmarking of VLMs under Stress
2026-08-07 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors created SABRE, a system that automatically builds tests to find weaknesses in vision-language models (VLMs). SABRE generates images and questions that challenge these models to see if they rely on actual visual clues or just familiar knowledge. When tested on six VLMs, the models scored poorly, showing they struggle with these tricky cases. The authors show that SABRE can be adapted for different types of difficult tests, making it a tool for ongoing evaluation rather than a one-time check.
vision-language modelsbenchmarkstress testimage generationquestion-answeringfiltering modelcontextattributecounterfactualannotation
Authors
Zixuan Lan, Luzhe Sun, Matthew R. Walter, Jiawei Zhou
Abstract
Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors -- learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark.