Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
2026-07-13 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors develop a mathematical framework to help explain how Transformer language models learn to reason inductively, which means making generalizations from examples. Instead of focusing on specific tasks, they study a broad group of related problems and show that the model's learning can be understood through a few simple, understandable factors rather than millions of parameters. Their approach also clarifies how the model's initial setup and data influence which reasoning strategies it develops. This framework helps identify what the model has learned and moves toward predicting how Transformers acquire reasoning abilities.
Transformer modelsinductive reasoningin-context learningattention mechanismslearning dynamicslow-dimensional manifoldmulti-hop reasoningmodel initializationcircuit formation
Authors
Tiberiu Musat, Tiago Pimentel, Nicholas Zucchet, Thomas Hofmann
Abstract
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are captured by a handful of interpretable coordinates rather than millions of parameters, making both theoretical and empirical analysis more tractable. Using this framework, we characterize how data statistics govern the competition between in-context and in-weights learning, we study how random initializations determine the `winning' circuit when multiple solutions are possible, and we demonstrate that the coordinate frame associated with the manifold can be used to automatically detect which circuits have been learned in trained models. By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.