KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models
2026-06-02 • Machine Learning
Machine LearningInformation Theory
AI summaryⓘ
The authors study how different vision-language models like CLIP and SigLIP represent data differently, beyond just looking at how well they perform tasks. They create a method called KODA that finds groups of examples that are clear in one model's view but not in another's, helping to understand structural differences. KODA uses kernels, a mathematical tool, to compare and align these models by identifying meaningful differences in how they cluster data. They also make their approach efficient for large datasets with random projection techniques. Their experiments show that KODA reveals consistent and understandable differences between these representations.
Vision-language modelsCLIPSigLIPContrastive learningKernel methodsRepresentation alignmentRandom Fourier FeaturesMultimodal learningDiscrepancy analysisDimensionality reduction
Authors
Youqi Wu, Mohammad Jalali, Farzan Farnia
Abstract
Vision-language foundation models such as CLIP and SigLIP provide widely used representations for multimodal learning systems. While these models are typically compared through downstream performance, such evaluations often do not explain how their representations differ structurally. In this work, we study this problem through the task of Contrastive Embedding Clustering: identifying sample subsets that are weakly clustered under one representation but strongly clustered under another. We propose \emph{Kernel Optimization for Discrepancy Analysis (KODA)}, a kernel-based framework for contrastive representation comparison and alignment. KODA constructs unified multimodal kernels through modality-wise kernel composition and formulates discrepancy discovery as a constrained optimization problem that searches for coherent structures in one representation while suppressing coherence in a reference representation. This yields interpretable discrepancy directions associated with specific sample subsets and modality interactions. To scale KODA to large vision-language datasets, we develop randomized low-dimensional approximations of joint kernels using random projections, including Random Fourier Features for shift-invariant kernels. Empirically, KODA identifies consistent and interpretable discrepancy structures across vision-language representations and provides sample subsets for representation alignment. The code is available at https://github.com/yokiwuuu/KODA.