Evidence-Backed Video Question Answering
2026-07-13 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors look at how video-based AI models answer questions but do so without clearly showing which parts of the video support their answers. They introduce a new task called E-VQA that makes models provide both an answer and detailed visual evidence from the video, like specific time segments and object masks. To help with this, they created a new benchmark dataset with human-verified visual grounding and a large-scale dataset to train models better at linking answers to precise video content. Their experiments show that just making models bigger doesn't solve the problem, but training with their data improves the models' ability to explain their answers visually.
Video Large Language ModelsQuestion AnsweringVisual GroundingSpatio-temporal SegmentationObject MaskingExplainabilityBenchmark DatasetFine-tuningVideo Understanding
Authors
Shijie Wang, Honglu Zhou, Ziyang Wang, Ran Xu, Caiming Xiong, Silvio Savarese, Chen Sun, Juan Carlos Niebles
Abstract
Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.