Staleness-Learning Rate Scaling Laws for Asynchronous RLHF
2026-07-01 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study how delays in using old data (stale rollouts) affect a type of reinforcement learning called asynchronous GRPO. They show that using outdated data causes a consistent error in the learning updates that depends on how old the data is and the learning rate. They also find that the stability of learning depends on both the total learning steps and the staleness, leading to specific conditions on how fast the learning rate can be. Their work explains why sometimes learning speed seems only lightly affected by data staleness.
Reinforcement LearningReward Learning from Human Feedback (RLHF)Asynchronous GRPOSurrogate ObjectiveStale RolloutsLearning RateStability ConditionsGradient BiasPolicy OptimizationBehavior Policy
Authors
Jingwei Song, Haofeng Xu, Jie Xiao, Chengke Bao, Jingwei Shi, Pengbin Feng, Weixun Wang, Yuhang Han, Chuan Wu, Linfeng Zhang, Bill Shi
Abstract
High-throughput RLHF systems often decouple rollout generation from policy optimization, leading to the use of stale rollouts during learner updates. In this work, we study the effect of such staleness in asynchronous GRPO. We make the behavior policy explicit in the GRPO surrogate objective and distinguish between the surrogate-gradient mapping used by the learner and the true total derivative of a distribution-dependent population objective. Under assumptions of local boundedness, distributional smoothness, and behavior-policy smoothness, we show that stale rollouts introduce a per-step surrogate-gradient bias of order O(S * eta), where S denotes the maximum rollout lag and eta denotes the learning rate. We further derive a conditional collapse-time scaling law: when within-cycle drift remains below a batch-level clipping radius, collapse is governed primarily by cumulative learner drift T * eta; when the stale-rollout constraint is active, stability instead depends explicitly on S * eta. This yields a two-constraint stability condition eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)}, explaining why the maximum stable learning rate may appear weakly dependent on staleness in the horizon-limited regime.