DSSMs: State Space Models with Explicit Memory via Delay Differential Equations
2026-07-11 • Machine Learning
Machine Learning
AI summaryⓘ
The authors introduce Delay State Space Models (DSSMs), an improved version of State Space Models designed to remember and use information from the past more accurately. Unlike regular SSMs that summarize all past data into a fixed-size memory, DSSMs add a way to explicitly include older states using ideas from delay differential equations. They develop new math and computational tricks to make this practical and efficient, especially using frequency domain techniques. Their tests show that DSSMs are better at tasks requiring recall of delayed information and perform well on standard sequence modeling benchmarks.
State Space ModelsDelay Differential EquationsDelayed FeedbackRecurrent ModelsFrequency DomainFast Fourier TransformLong-Sequence ModelingKernel MethodsParameterizationSequence Metrics
Authors
Yixiao Qian, Song Chen, Jiaxu Liu, Shengze Cai, Chao Xu
Abstract
State Space Models (SSMs) have emerged as a powerful paradigm for efficient long-sequence modeling, offering parallel training and fast linear-time recurrent inference. However, like other recurrent architectures, SSMs must compress an unbounded history into a fixed-size state, which limits context retention and makes precise retrieval over long-range context inherently difficult. To overcome this limitation, we propose Delay State Space Models (DSSMs), a delay differential equation (DDE)-inspired extension of diagonal SSMs that augments discrete SSM recurrences with explicit delayed-state feedback. Making explicit delayed feedback practical requires new stability parameterization, history management, and FFT-training tools. We address these challenges with a practical discretization and parameterization grounded in a simple delay-independent stability condition. To bypass direct time-domain kernel construction, we derive the DSSM transfer function and compute kernels in the frequency domain, using a kernel contour shift to suppress aliasing and recover accurate FFT training. Empirically, DSSMs substantially improve targeted delayed-retrieval tasks while outperforming S4D on most standard sequence metrics and remaining close on the others.