Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
2026-08-04 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors explain that large language models improve problem-solving by using different methods during inference, like following a single path or exploring many possibilities. They show that these methods vary a lot and can't be compared easily without clear rules. To fix this, the authors create a system to categorize and evaluate these inference methods in a fair way and highlight what details must be shared to reproduce results. They apply their approach to various reasoning tasks and provide a large dataset of reasoning examples for others to use.
large language modelsinference-time computetest-time scalingautoregressive modelsinference algorithmsprefix treeevaluation metricsreproducibilitysymbolic reasoningbenchmark
Authors
Mohsen Hariri, Weicong Chen, Nahal Shahini, Vikash Singh, Kai Ye, Amirhossein Samandar, Debargha Ganguly, Sreehari Sankar, Yanyan Zhang, Shouren Wang, Jerry Peng, Biyao Zhang, Michael Hinczewski, Vipin Chaudhary
Abstract
Large language models can solve substantially harder reasoning problems with more inference-time compute. The term "test-time scaling," however, now covers diverse inference algorithms that extend deliberation along a single trajectory, sample completed candidates and aggregate them through voting or verification, or search over unfinished partial states. These algorithms differ in their statistical structure, compute accounting, and failure modes. Treating these procedures as interchangeable under a single scalar "budget," or reporting accuracy without the inference protocol that produced it, makes results difficult to compare across studies. We develop a systematic account of test-time scaling along three axes. First, we formalize test-time scaling as budgeted inference over the implicit prefix tree of an autoregressive model and distinguish three structural regimes: single-trajectory sequential scaling, leaf-level scaling with terminal reduction, and prefix-level scaling. Second, we treat the evaluated object as the entire inference system and develop evaluation principles that separate end-to-end system performance from candidate-bank diagnostics. We introduce an evaluation profile whose coordinates and simple functionals recover or bound common repeated-sampling metrics, and prescribe protocol-matched reporting of compute and uncertainty. Third, we specify reproducibility requirements for inference protocols, distinguishing exact replay from distributional reproducibility and identifying the artifacts needed to support each. We also organize the open-weight reasoning ecosystem by model-side and interface mechanisms, apply these principles to broad-knowledge, symbolic-reasoning, and competition-mathematics benchmarks, and assemble over 2 billion full reasoning traces for release with progressively richer verifier and token-level signals.