RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval
2026-04-10 • Computation and Language
Computation and LanguageArtificial IntelligenceInformation RetrievalMachine Learning
AI summaryⓘ
The authors propose RecaLLM, a new type of language model designed to better use very long texts by switching back and forth between thinking steps and looking up important information. They found that when a model thinks too long without checking back, it gets worse at finding needed details, a problem they call lost-in-thought. RecaLLM fixes this by alternating reasoning with explicit retrieval of relevant facts, helping it stay accurate even with very long inputs. Their experiments show that this method works well on long-text benchmarks, even when trained with shorter examples than usual.
language modelslong-contextin-context retrievalreasoningconstrained decodingRULER benchmarkHELMET benchmarktext generationcontext windowtraining data
Authors
Kyle Whitecross, Negin Rahimi
Abstract
We propose RecaLLM, a set of reasoning language models post-trained to make effective use of long-context information. In-context retrieval, which identifies relevant evidence from context, and reasoning are deeply intertwined: retrieval supports reasoning, while reasoning often determines what must be retrieved. However, their interaction remains largely underexplored. In preliminary experiments on several open-source LLMs, we observe that in-context retrieval performance substantially degrades even after a short reasoning span, revealing a key bottleneck for test-time scaling that we refer to as lost-in-thought: reasoning steps that improve performance also make subsequent in-context retrieval more challenging. To address this limitation, RecaLLM interleaves reasoning with explicit in-context retrieval, alternating between reasoning and retrieving context information needed to solve intermediate subproblems. We introduce a negligible-overhead constrained decoding mechanism that enables verbatim copying of evidence spans, improving the grounding of subsequent generation. Trained on diverse lexical and semantic retrieval tasks, RecaLLM achieves strong performance on two long-context benchmarks, RULER and HELMET, significantly outperforming baselines. Notably, we observe consistent gains at context windows of up to 128K tokens using training samples of at most 10K tokens, far shorter than those used by existing long-context approaches, highlighting a promising path toward improving long-context performance without expensive long-context training data.