Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors address the problem that captions describing images often miss some details, which can confuse models trying to match pictures and words. They created a new method called Text as Partial Constraint (TPC) that focuses on the most important shared meaning in different captions, ignoring parts that aren’t clearly stated. Their system also learns to be less confident when captions disagree, making it more reliable. Tests show TPC improves image recognition and language understanding tasks, especially when captions are incomplete or tricky.
Vision-language alignmentOpen-vocabulary recognitionContrastive learningZero-shot recognitionAdversarial robustnessText as Partial Constraint (TPC)Language-vision models (LVLM)Uncertainty-aware learningCaption underspecification
Authors
Chengzhen Yu, Canran Xiao, Siyuan Ma, Yang Liu
Abstract
Vision-language alignment powers open-vocabulary recognition, retrieval, and LVLM grounding, yet natural captions are often underspecified, making similarity brittle and overly confident under paraphrase and omitted details. We aim to learn representations whose matching is stable across caption views and whose confidence reflects how strongly text constrains an image. We propose Text as Partial Constraint (TPC), a core-residual alignment framework that treats multi-view captions as incomplete supervision. It distills a consensus semantic core as the alignment target, learns a single-view core predictor for standard inference with one query, and explicitly discourages vision-language similarity from depending on the orthogonal unsaid residual. An uncertainty-aware contrastive objective further softens alignment when caption views disagree, reducing overconfident updates under weak language constraints. Across zero-shot recognition and adversarial robustness, TPC achieves 81.42/64.05 Top-1 clean/robust accuracy on ImageNet and 76.19/52.03 on an Avg-14 transfer suite, while improving LVLM transfer with 85.16 POPE F1 and 59.57 OKVQA accuracy under an LLaVA-1.5-7B stack. These results suggest that modeling text as a partial constraint is a practical and principled route to more reliable vision-language representations under underspecified language supervision.