Contravariance Theory: Strong Alignment for Minimal Solutions to Hard Tasks
2026-07-09 • Machine Learning
Machine Learning
AI summaryⓘ
The authors investigate how different deep neural networks (DNNs) that solve the same difficult task tend to develop similar internal structures. They show that even a loose alignment between the networks' representations leads to strong alignment of important internal directions, which become clearer as you look deeper into the network layers. This means that when networks are trained end-to-end on complex tasks, they naturally evolve similar features. Their work suggests that the way we compare these networks is less critical than previously thought, and that it is likely networks will converge to similar solutions over time.
Deep Neural NetworksNeuroAIRepresentation AlignmentAffine MappingConvergent EvolutionPrivileged AxesHierarchy in Neural NetworksTask OptimizationCovariance and Contravariance theory
Authors
Dan Yamins, Aran Nayebi
Abstract
A series of results from the NeuroAI over the past fifteen years have raised core questions both about how to compare Deep Neural Network (DNN) models to the brain, and about how much convergent evolution to expect between artificial networks and real brain networks. Here, we show that for any two minimal DNN solutions to a sufficiently hard task: (i) "weak" alignment of network representations based on affine mappings guarantees "strong" alignment of privileged axes, and (ii) alignment "zippers" up the network hierarchy, causing the emergence of privileged axes from end-to-end task optimization. These results formalize the notion of contravariance from Cao and Yamins [2024], and illustrate important consequences for the theory of NeuroAI: with sufficiently strong tasks, choice of metric for inter-network comparison is not all that sensitive, and that convergent evolution is probably inevitable.