Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks
2026-09-03 • Machine Learning
Machine LearningNeural and Evolutionary Computing
AI summaryⓘ
The authors study a type of neural network that processes information over time, which can struggle with delays and weakening signals in deep layers. They develop Recursive Quadrature Filters (RQFs), a biologically inspired method that acts like a tunable band-pass filter to improve memory in these networks. By adjusting the way bottom-up inputs are handled, they reduce problems caused by depth, especially when only partial learning signals are used. Their experiments show that this approach helps different models perform better or the same with fewer parameters. Overall, the authors highlight RQFs as efficient tools for memory in recurrent networks and propose a fix to handle deep network input issues.
temporal integrationcontinuous-time recurrent networksRecursive Quadrature Filtersstate-space modelsband-pass filterbackpropagation through timegradient attenuationspeech commands datasetPath-X taskprospective-input coding
Authors
Shivang Rawat, Mirko Morello, Flaviano Morone, David J. Heeger
Abstract
Temporal integration gives continuous-time recurrent networks memory, but in deep stacks it also delays bottom-up signals and attenuates top-down errors. We develop Recursive Quadrature Filters (RQFs), biologically motivated complex-valued temporal filters that are a special case of diagonal state-space models (SSMs), and ask whether this failure mode can be addressed by making each layer's bottom-up input prospective. Starting from an energy model, we derive the RQF dynamics and show that each RQF is a band-pass filter whose learnable parameters control its tuning frequency and bandwidth. We then make each layer's bottom-up input prospective using a parameter-free two-tap update that leaves the recurrent transition and parallel scan unchanged. We extend this correction to general diagonal SSMs and show that it mitigates depth-dependent gradient attenuation when temporal gradients are truncated, i.e., spatial-only backpropagation. We evaluate the intervention in RQFs, S5, and ORGaNICs (a nonlinear gated RNN) trained using full backpropagation through time (BPTT) and spatial-only backpropagation. Under full BPTT, prospective variants match or outperform their non-prospective controls in every model and configuration. A non-residual width-32 six-layer RQF reaches 96.09% accuracy on raw-audio Speech Commands with 31.9k parameters; a width-64 six-layer RQF reaches 83.56% on the 16,384-step Path-X task. These results identify RQFs as a parameter-efficient recurrent substrate and prospective-input coding as an input-side correction for deep continuous-time recurrent networks.