CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation
2026-06-30 • Machine Learning
Machine LearningComputer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of measuring uncertainty in AI models that handle both text and images, known as multimodal large language models (MLLMs). They propose a method called CoMet that breaks down uncertainty into two parts: one related to the task or prompt (context), and another related to how many different valid answers there could be (multiplicity). Their approach uses a lightweight module to estimate uncertainty efficiently, without needing to generate many answers repeatedly. Tests show CoMet improves how well the model knows when it might be wrong, across various tasks involving open-ended questions and image understanding.
Uncertainty EstimationMultimodal Large Language ModelsContext-Specific UncertaintyMultiplicity-Specific UncertaintyPost-hoc ModuleHallucination DetectionOpen-ended Question AnsweringVisual Question AnsweringAutoregressive Generation
Authors
Sanghyuk Chun, William Yang, Amaya Dharmasiri, Olga Russakovsky
Abstract
Uncertainty estimation has been a long-standing challenge in AI models; it amounts to "knowing what you don't know," and metacognition is notoriously difficult even for humans (cf. the Dunning-Kruger effect). Although it is still far from solved even in simpler classification systems, tackling it in multimodal large language models (MLLMs) is becoming increasingly important. Within MLLMs, uncertainty can stem from any of the diverse sources as well as from their relationships, and further can stem from the unbounded answers in the open-ended setting. To tackle the issues, we propose CoMet, an MLLM uncertainty estimation method by decomposing uncertainty into a context-specific term and a multiplicity-specific term. The former captures ambiguity induced by the given context (e.g., task or prompt), while the latter captures how many plausible answers determined by the context remain compatible with the given input. We train a lightweight post-hoc uncertainty module to estimate these quantities, which enables efficient uncertainty estimation without autoregressive answer generation or repeated sampling. Experiments on various open-ended multimodal benchmarks, hallucination detection, and multiple-choice visual question answering benchmarks show that CoMet consistently improves uncertainty estimation over existing baselines while remaining efficient in practice. Code is available at https://github.com/princetonvisualai/comet_uncertainty