On Subquadratic Architectures: From Applications to Principles
2026-06-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors compare three types of sequence models designed to be faster than traditional transformers, called xLSTM, Mamba-2, and Gated DeltaNet, across tasks like code modeling and time-series analysis. They find that xLSTM performs best overall, mainly because its design allows it to better track and update memory over time. By analyzing the models, the authors explain that xLSTM's gating system gives it more flexible and stable memory correction, which helps in handling complex dependencies. They also confirm these advantages through controlled tests on synthetic data. In short, the authors show that xLSTM's strength comes from how well it manages internal state and remembers information.
TransformersAttention MechanismSubquadratic ArchitecturesxLSTMGated DeltaNetMemory DynamicsState TrackingSequence ModelingCode Model Pre-trainingTime-Series Foundation Models
Authors
Anamaria-Roberta Hartl, Levente Zólyomi, David Stap, Pieter-Jan Hoedt, Niklas Schmidinger, Lukas Hauzenberger, Sebastian Böck, Günter Klambauer, Sepp Hochreiter
Abstract
Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from large language models, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.