Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search
2026-08-20 • Information Retrieval
Information Retrieval
AI summaryⓘ
The authors improved a benchmark that tests how well AI agents can search for answers by using a much larger, more realistic document collection instead of a small fixed set. They created a system that breaks down questions into smaller reasoning steps and finds evidence for each step in this big dataset. Their method keeps only questions that are fully supported by the new collection after automatic and human checks. This makes the task harder because it depends more on good retrieval, and even strong AI agents find it more challenging. They also share their new benchmark and tools online.
benchmarkagentic searchcorpusretrievalreasoning hopsevidence groundingClimbMixquestion verificationnatural language processinglanguage models
Authors
Sahel Sharifymoghaddam, Lingwei Gu, Yijun Ge, Jimmy Lin
Abstract
The BrowseComp-Plus benchmark disentangled the evaluation of agentic search by replacing opaque web search with a fixed corpus, so that an agent's role can be separated from the retriever's. That corpus, however, holds only about 100K documents and was assembled from the supporting documents of the benchmark's own queries plus mined hard negatives, so the evidence and the distractors were both selected per query. We introduce $\text{BrowseComp-Plus}_{\text{CM}}$, which keeps the BrowseComp-Plus questions but relocates their evidence to ClimbMix, a 400B-token, 553M-document mixture of web text released by NVIDIA for pre-training language models and built without reference to any benchmark. Our main contribution is the projection pipeline that makes this possible: it decomposes each question into atomic reasoning hops and grounds every hop in the new corpus, retaining a question only when automatic verification, an independent agent, and human review all confirm that every hop is supported. The pipeline is dataset-agnostic and applies to any benchmark whose questions decompose into verifiable facts. Applied to the 830 BrowseComp-Plus test questions, our pipeline yields 57 fully grounded questions with question-level relevance judgments. Projection shifts the difficulty onto retrieval, as the strongest agent we evaluate loses five points of answer accuracy but sees its evidence recall fall from 84.3% to 21.4% while issuing 63% more search calls. As the first of a series of projections, we release the pipeline, the benchmark, and our analyses at https://github.com/castorini/cmass.