Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
2026-04-02 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study ways to improve learning in large language models after they have been trained, focusing on how to assign credit for correct or incorrect outputs. They find that existing methods either penalize too broadly or improve early but become unstable later. To fix this, the authors create a new method called Sample-Routed Policy Optimization (SRPO) that treats correct and incorrect outputs differently and uses a system to focus on reliable feedback. Their approach both learns quickly and remains stable over time, outperforming older methods on several tests while being more efficient. This shows a better balance between fast improvement and stable training.
Reinforcement LearningLarge Language ModelsPolicy OptimizationCredit AssignmentSelf-DistillationGRPOSDPOEntropyOn-policy LearningLogit-level Supervision
Authors
Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, Tat-Seng Chua
Abstract
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address specific deviations. Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision that facilitates rapid early improvement, yet it frequently collapses during prolonged training. We trace this late-stage instability to two intrinsic flaws: self-distillation on already-correct samples introduces optimization ambiguity, and the self-teacher's signal reliability progressively degrades. To resolve these issues, we propose Sample-Routed Policy Optimization (SRPO), a unified on-policy framework that routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction. SRPO further incorporates an entropy-aware dynamic weighting mechanism to suppress high-entropy, unreliable distillation targets while emphasizing confident ones. Evaluated across five benchmarks and two model scales, SRPO achieves both the rapid early improvement of SDPO and the long-horizon stability of GRPO. It consistently surpasses the peak performance of both baselines, raising the five-benchmark average on Qwen3-8B by 3.4% over GRPO and 6.3% over SDPO, while simultaneously yielding moderate response lengths and lowering per-step compute cost by up to 17.2%.