Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models
2026-02-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors study how intelligent systems can recognize familiar parts when they appear in new combinations, a skill called compositional generalization. They identify three important properties (divisibility, transferability, stability) and show that to achieve these, representations need to break down linearly into separate, nearly-orthogonal parts corresponding to concepts. This explains why neural network representations often have a linear structure. They test this idea with vision models and find partial evidence supporting their theory, noting that better linear separation relates to better generalization on new concept mixes.
Compositional generalizationLinear Representation HypothesisOrthogonalityEmbedding geometryNeural representationsVision modelsConcept decompositionLow-rank factorization
Authors
Arnas Uselis, Andrea Dittadi, Seong Joon Oh
Abstract
Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems. Although modern models are trained on massive datasets, they still cover only a tiny fraction of the combinatorial space of possible inputs, raising the question of what structure representations must have to support generalization to unseen combinations. We formalize three desiderata for compositional generalization under standard training (divisibility, transferability, stability) and show they impose necessary geometric constraints: representations must decompose linearly into per-concept components, and these components must be orthogonal across concepts. This provides theoretical grounding for the Linear Representation Hypothesis: the linear structure widely observed in neural representations is a necessary consequence of compositional generalization. We further derive dimension bounds linking the number of composable concepts to the embedding geometry. Empirically, we evaluate these predictions across modern vision models (CLIP, SigLIP, DINO) and find that representations exhibit partial linear factorization with low-rank, near-orthogonal per-concept factors, and that the degree of this structure correlates with compositional generalization on unseen combinations. As models continue to scale, these conditions predict the representational geometry they may converge to. Code is available at https://github.com/oshapio/necessary-compositionality.