Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams
2026-07-11 • Machine Learning
Machine LearningArtificial IntelligenceComputation and LanguageInformation Retrieval
AI summaryⓘ
The authors study how models remember information by focusing on the unique pieces of data rather than every single token in a sequence. They propose a memory system that only saves new, distinct information in a special cache, combined with other memory parts that handle summaries and recent details. Their experiments show this method can perform as well as traditional full-attention models while using less memory, especially when dealing with long sequences. This approach also makes the memory easier to understand and inspect. The work is preliminary and tested on small, public datasets.
context engineeringtokenscachecontent-addressable memorynovelty gatingrecurrent statesliding windowattention mechanismnext-code predictionstate-space summary
Authors
Siddharth Pal, Viktoria Rojkova
Abstract
Context engineering decides what information a model carries forward, and current designs meter it in tokens: compressing the past into a bounded recurrent state, keeping a key-value entry for every token, or imposing a fixed budget through a window or eviction rule. All three make the token the unit of memory even when the stream is redundant and the task depends on the distinct information it carries. Building on a companion mechanism paper that opens a cache slot only when an incoming key is novel, so memory scales with the number of distinct items rather than tokens, we develop that allocate-on-novelty cache as a working-memory component and organize context by how a task depends on the past: recall-carried information belongs in a content-addressed novelty cache, summary-carried information in a recurrent state, and locality-carried information in a recency window. The claim is empirical and bounded. On a matched character-level control, novelty-gated attention reaches full-attention performance while attending to about half the tokens, and coupling the cache with a state-space summary matches full-attention coupling at that reduced cost; the advantage grows as context lengthens, while a sliding window is preferable on short, locality-dominated spans. On next-code prediction over synthetic Medicare claims the coupled component leads full attention and every fixed-budget eviction policy at a thousand-event horizon, whereas cost forecasting over the same stream is summary-carried and the cache is neutral. The retained memory is an inspectable table of templates, codes, drugs, or places rather than an opaque state. The experiments are small-scale and use only public data; they establish the primitive that context can scale with distinct information rather than tokens, in a working memory that is content-addressable and auditable.