LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards
2026-05-29 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors address the challenge of long-context reasoning in large language models, where it is hard to find and use important information from lots of distracting text. They create a new method called LongTraceRL that builds tricky training examples with different levels of distracting information and provides more detailed rewards for reasoning steps using a 'rubric reward.' This helps models learn better how to reason step-by-step when the final answer is correct. Their experiments show that LongTraceRL improves performance on various tests compared to previous methods.
long-context reasoninglarge language modelsreinforcement learningknowledge graphmulti-hop questionsdistractorsreward designrubric rewardprocess supervisionsearch trajectories
Authors
Nianyi Lin, Jiajie Zhang, Lei Hou, Juanzi Li
Abstract
Long-context reasoning remains a central challenge for large language models, which often fail to locate and integrate key information in extensive distracting content. Reinforcement learning with verifiable rewards (RLVR) has shown promise for this task, yet existing methods are limited by low-confusability distractors and sparse, outcome-only reward signals that cannot supervise intermediate reasoning steps. To address these issues, we introduce \textsc{LongTraceRL}. For data construction, we generate multi-hop questions via knowledge graph random walks and leverage search agent trajectories to build \emph{tiered distractors}: documents the agent read but did not cite (high confusability) and documents that appeared in search results but were never opened (low confusability), producing training contexts that are far more challenging than those built by random sampling or one-shot search. For reward design, we propose a \emph{rubric reward} that uses the gold entities along each reasoning chain as fine-grained, entity-level process supervision. This rubric reward is applied only to responses with correct final answers (positive-only strategy), distinguishing the reasoning quality among correct responses and preventing reward hacking. Experiments on three reasoning LLMs (4B--30B) across five long-context benchmarks demonstrate that \textsc{LongTraceRL} consistently outperforms strong baselines and encourages comprehensive, evidence-grounded reasoning. Codes, datasets and models are available at \href{https://github.com/THU-KEG/LongTraceRL}{https://github.com/THU-KEG/LongTraceRL}.