Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
2026-06-16 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied looped architectures, which help models learn tasks step-by-step, but can suffer from problems as the number of steps increases. They fixed this by using certain design tricks called pre-norm layers and residual scaling to keep the signal strong. Then, they created a Transformer-based model called FPRM that decides when to stop looping by finding a fixed point, so it works efficiently depending on how hard the task is. They tested this model on several reasoning problems like Sudoku and mazes and found it works well.
Looped architecturesSignal propagationPre-norm layersResidual scalingTransformerFixed-point convergenceHalting mechanismCompositional reasoningSudoku benchmarkReasoning tasks
Authors
Sajad Movahedi, Vera Milovanović, Shlomo Libo Feigin, Alexander Theus, Thomas Hofmann, Valentina Boeva, T. Konstantin Rusch, Antonio Orvieto
Abstract
Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by looping determines the quality of the solution these models find. Like deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address this signal propagation issue using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM, a Transformer-based Fixed-Point Reasoning Model that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to task difficulty. FPRM is effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking, and ARC-AGI.