Skip-Connected Policy Optimization for Implicit Advantage
2026-04-09 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors look at improving how reinforcement learning optimizes reasoning by using rewards given at different steps. They find that trying to use detailed rewards at every step often causes noisy and inconsistent learning early on, which can hurt performance. To fix this, they introduce Skip-Connected Optimization (SKPO), which splits reasoning into two parts: one part learns from detailed feedback, while the other part keeps a safer group-based optimization and can skip faulty reasoning when needed. Their tests show SKPO improves results on math and reasoning tasks and produces better step-by-step reasoning even when final answers are correct.
Reinforcement LearningGroup Relative Policy OptimizationMonte Carlo EstimationDense RewardsSkip-Connected OptimizationReasoning TokensOptimizationTrajectory QualityIntermediate RewardsMathematical Benchmarks
Authors
Fengwei Teng, Jinyi Bai, Xinhao Yao, Demi Ruohan Wang, Jiahao Zhao, Zhijiang Guo
Abstract
Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness.