Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
2026-02-19 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study how asynchronous training in reinforcement learning can make learning unstable due to high variability in the policy updates. They find that when training uses outdated data, a few samples can overly influence updates, making the gradients noisy and causing model collapse. To fix this, they propose VCPO, a method that adjusts learning rates based on sample quality and uses a special baseline to reduce variance without extra complexity. Tests show VCPO makes asynchronous training more reliable and faster while maintaining good performance.
reinforcement learningasynchronous trainingpolicy gradientREINFORCEvariance reductioneffective sample sizebaselineoff-policy learninggradient stabilitylarge language models
Authors
Luke Huang, Zhuoyang Zhang, Qinghao Hu, Shang Yang, Song Han
Abstract
Reinforcement learning (RL) is widely used to improve large language models on reasoning tasks, and asynchronous RL training is attractive because it increases end-to-end throughput. However, for widely adopted critic-free policy-gradient methods such as REINFORCE and GRPO, high asynchrony makes the policy-gradient estimator markedly $\textbf{higher variance}$: training on stale rollouts creates heavy-tailed importance ratios, causing a small fraction of samples to dominate updates. This amplification makes gradients noisy and learning unstable relative to matched on-policy training. Across math and general reasoning benchmarks, we find collapse is reliably predicted by effective sample size (ESS) and unstable gradient norms. Motivated by this diagnosis, we propose $\textbf{V}$ariance $\textbf{C}$ontrolled $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{VCPO}$), a general stabilization method for REINFORCE/GRPO-style algorithms that (i) scales learning rate based on effective sample size to dampen unreliable updates, and (ii) applies a closed-form minimum-variance baseline for the off-policy setting, avoiding an auxiliary value model and adding minimal overhead. Empirically, VCPO substantially improves robustness for asynchronous training across math, general reasoning, and tool-use tasks, outperforming a broad suite of baselines spanning masking/clipping stabilizers and algorithmic variants. This reduces long-context, multi-turn training time by 2.5$\times$ while matching synchronous performance, demonstrating that explicit control of policy-gradient variance is key for reliable asynchronous RL at scale.