Phantom Gains: Auditing Self-Improvement Against a Measured Null
2026-08-20 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors examine how to accurately track improvements in language models by looking at individual problem gains and losses, rather than average performance. They find that common methods for measuring these changes can be misleading due to noise and artifacts, especially when lacking proper controls. They propose a new statistical test that better accounts for expected variations using baseline replicates, which shows that self-training often does not improve performance as previously thought, while distillation does. Their work highlights the importance of careful measurement and null models when auditing model improvements on specific tasks.
language modelself-trainingdistillationLoRAmeasurement noisenull hypothesis testingfalse discovery ratebaseline replicatesmodel evaluation
Authors
Cheng Xu, Nan Yan, Liming Chen, M-Tahar Kechadi
Abstract
Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank-$32$ LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of $0.280$. The natural threshold repair does not survive replication: estimated across the frozen comparisons such a design already contains, its null stays non-zero. We replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control, which detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size. Applied to a ladder of arms matched in stream, volume and evaluation, the audit finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not; a regression rejects this asymmetry as a by-product of distillation's larger overall gain ($p < 10^{-8}$). On the far smaller set of problems the base model never reaches, the evidence is inconclusive, while self-training corrupts problems solved at baseline at rates well above the measured floor. Transition-level auditing therefore requires a separately measured null for every statistic it reports: nulls that cost no new experiments, built from baseline replicates a multi-arm study already owns, though not from as few as most possess.