Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation
2026-04-09 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on a task where a computer needs to find and label any object in a picture based on text descriptions. Usually, this requires complex training that matches picture details to words bit by bit. Instead, the authors discovered a way to skip this slow training by solving an equation directly that links image parts and words. They noticed that parts of the image showing the same thing have similar differences when matched with words, and they use this idea to label images quickly and accurately. Their method works well across multiple test sets without extra complicated steps.
Open-vocabulary semantic segmentationVision-language alignmentCosine similarityLogitsDistribution discrepancyPixel-level segmentationAnalytic solutionIterative trainingBenchmark datasets
Authors
Jiahao Li, Yang Lu, Yachao Zhang, Fangyong Wang, Yuan Xie, Yanyun Qu
Abstract
Open-vocabulary semantic segmentation (OVSS) aims to segment arbitrary category regions in images using open-vocabulary prompts, necessitating that existing methods possess pixel-level vision-language alignment capability. Typically, this capability involves computing the cosine similarity, \ie, logits, between visual and linguistic features, and minimizing the distribution discrepancy between the logits and the ground truth (GT) to generate optimal logits that are subsequently used to construct segmentation maps, yet it depends on time-consuming iterative training or model-specific attention modulation. In this work, we propose a more direct approach that eschews the logits-optimization process by directly deriving an analytic solution for the segmentation map. We posit a key hypothesis: the distribution discrepancy encodes semantic information; specifically, this discrepancy exhibits consistency across patches belonging to the same category but inconsistency across different categories. Based on this hypothesis, we directly utilize the analytic solution of this distribution discrepancy as the semantic maps. In other words, we reformulate the optimization of the distribution discrepancy as deriving its analytic solution, thereby eliminating time-consuming iterative training, freeing us from model-specific attention modulation, and achieving state-of-the-art performance on eight benchmark datasets.