AI summaryⓘ
The authors study looped transformers, which try to solve harder problems by repeating computations more times during testing. They develop a framework based on fixed-point theory to understand when these repeated computations produce reliable and meaningful results instead of just memorizing training examples. Their math shows that adding a mechanism called recall and a technique called outer normalization helps the model reach stable and smooth solutions that depend on the input. They test their ideas on tasks like chess and sudoku and find that their theory matches how well the models perform. They also create a new way to use recall internally, which works even better on sudoku when combined with outer normalization.
looped transformersfixed-point iterationrecallouter normalizationinput-dependencereachabilitystable backpropagationchess AIsudoku solvingprefix sums
Abstract
Looped transformers promise test-time compute scaling by spending more iterations on harder problems, but it remains unclear which architectural choices let them extrapolate to harder problems at test time rather than memorize training-specific solutions. We introduce a fixed-point based framework for analyzing looped architectures along three axes of stability -- reachability, input-dependence, and geometry -- and use it to characterize when fixed-point iteration yields meaningful predictions. Theoretically, we prove that looped networks without recall have countable fixed points and cannot achieve strong input-dependence at any spectral regime, while recall combined with outer normalization reliably produces a regime in which fixed points are simultaneously reachable, locally smooth in the input, and supported by stable backpropagation. Empirically, we train single-layer looped transformers on chess, sudoku, and prefix-sums and find that downstream performance tracks the framework's predictions across tasks and architectural configurations. We additionally introduce internal recall, a novel recall placement variant, and show that it becomes competitive with -- and on sudoku, substantially better than -- standard recall placement once outer normalization is applied.