SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models
2026-06-30 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors introduce Semantic Reference Frames (SemRF) to better understand how language models process information layer by layer without confusion from shifting measurement points. SemRF uses fixed anchors to provide a stable perspective, making it easier to track semantic changes across layers accurately. They analyze how these changes form paths with certain mathematical properties, linking simpler paths to more efficient model behaviors. The work includes methods to measure step contributions and handle noise, ensuring reliable semantic tracking through the network depth.
Residual-stream analysisSemantic Reference FramesLanguage modelsEmbeddingUnembeddingSemantic trajectoryVoronoi diagramQuadratic optimizationParameter efficiency
Authors
Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang
Abstract
Residual-stream analysis asks how language-model computation evolves across depth, but intermediate decoding requires comparable readout coordinates across layers. If embedding anchors and unembedding readout disagree on the chosen span, apparent motion may reflect measurement drift rather than computation. We introduce \emph{Semantic Reference Frames} (SemRF), an anchor-based formalism separating semantic measurement from residual dynamics. A SemRF fixes anchors and measures states against them. Pseudo-inverse tying gives exact synchronization; under restricted bi-invertibility, SemRF yields stable semantic-basis coordinates, distortion bounds, and near-identity changes. With the frame fixed, residual computation becomes a depthwise semantic trajectory. The anchors induce a semantic Voronoi diagram: distance, or evidence such as logits, assigns each layer to a coarse cell, while coordinates retain within-cell motion and margins. We define layerwise steps, contribution profiles, and imbalance diagnostics, then use the Voronoi trace to define a margin-relaxed tube. The canonical trace is the minimum-action path inside this tube; when nonempty with positive quadratic weight, it is unique and obeys a discrete spline equation away from active constraints. Excess action controls step, curvature, and profile mismatch. Low curvature implies piecewise-linear compressibility and local knowledge density: lower trace complexity means fewer semantic knots. Through the parameter-to-trajectory map, this gives a conditional link to parameter efficiency: among admissible settings fitting data, lower-action and lower-complexity traces use fewer semantic degrees of freedom. The guarantees require controlled interface error and small projection residual under explicit tube constraints.