Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
2026-08-17 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors explain that when models try to remember long sequences, they often use a fixed memory size which can get overwhelmed early on and hurt performance later. They propose a new way called incremental memory activation, where the model starts with a small memory and gradually increases it as it processes more information. This approach helps the model compress earlier information better and keep new information clearer. They tested this idea with several advanced models and found it improved performance, especially on longer sequences. Their work shows that dynamically adjusting memory capacity is a useful strategy for sequence tasks.
attention-based modelssequence modelingmemory compressionincremental memory activationcontext lengthlanguage modelingneural memory architectureslong-context retrievalmodel capacityinterference
Authors
Reza Bayat, Ali Behrouz, Vahab Mirrokni, Aaron Courville
Abstract
The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state. However, most existing memory models expose a static memory throughout the entire sequence. Because early tokens face no compression pressure, they occupy too many degrees of freedom and "pollute" the memory state, leaving little capacity for later context and increasing interference between what is stored and what arrives next. We study a new paradigm of incremental memory activation, where the effective capacity of memory is progressively expanded as the context grows. Imposing an early bottleneck forces the model to compress history more effectively, while unlocking fresh capacity over time reduces interference and improves retention of later context. We instantiate this paradigm in Proteus, a straightforward mechanism that can be incorporated into a broad class of neural memory architectures at no additional cost. We apply Proteus to state-of-the-art models, including SWLA, Comba, Titans, and Hope-Attention, and observe consistent improvements on standard language modeling and reasoning, as well as on long-context retrieval and understanding, with gains that grow at longer context lengths. Overall, our results show that static memory is suboptimal and that scheduling effective capacity is a simple and broadly applicable tool for sequence modeling.