From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers
2026-07-12 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors explain self-attention in Transformers using a new geometric viewpoint, treating token sequences like flows on a graph. They show that single-head attention acts like a simple constant connection step, while multi-head attention mixes these steps depending on the token connections. Their theory links attention to classical geometry, and they find that real Transformer models behave as their math predicts, especially in deeper layers and larger models. This work provides tools to better understand how attention works from a geometric perspective.
self-attentionTransformervector fieldgraph theoryconnection walkmulti-head attentionrandom walkLaplaciangeometryisometry
Authors
Binbin Lin, Wei Chen, Yalun Li, Wenxiao Wang, Jieping Ye, Xiaofei He
Abstract
Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood. We view a token sequence as a vector field over the token-position graph and identify attention as a connection walk: messages are aggregated by a nonnegative walk matrix while being transported along each edge by a learned linear map. Within this framework, we prove that single-head attention (SHA) is exactly a connection propagation step with constant transport, and that multi-head attention (MHA) is exactly a single edge-dependent connection walk whose effective transport is an attention-gated mixture of headwise transports. We further clarify the conditions under which the corresponding generator reduces to a random-walk connection Laplacian, highlighting the roles of stochasticity, reversibility, and metric-compatible transports. Empirically, we find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory: effective attention graphs converge to stable geometric operators in deeper layers, learned transports self-organize into approximate scaled isometries, and both phenomena strengthen consistently with scale. Overall, the paper provides a precise connection-walk formalism that links self-attention to classical geometric operators, along with a set of operator-level tools for analyzing transformer models from a geometric perspective.