Rethinking Exploration in RLVR: From Entropy Regularization to Refinement via Bidirectional Entropy Modulation
2026-04-06 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors study a problem in teaching AI language models to explore different ways of solving tasks, called "restricted exploration," where the model quickly sticks to a few solutions. They find that simply trying to keep things random (entropy) isn't enough and can even hurt learning. By breaking down this randomness into helpful parts (informative entropy) and harmful parts (spurious entropy), they introduce a new method called AsymGRPO that treats good and bad outcomes differently to better keep useful exploration. Their experiments show this method works better than existing approaches and can work alongside other techniques.
Reinforcement LearningLarge Language ModelsPolicy EntropyExplorationEntropy RegularizationAdvantage EstimationReward OptimizationAsymGRPORolloutsExploration-Exploitation Tradeoff
Authors
Hengrui Gu, Xiaotian Han, Yujing Bian, Kaixiong Zhou
Abstract
Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning capabilities of large language models (LLMs). However, it faces a fundamental limitation termed \textit{restricted exploration}, where the policy rapidly converges to a narrow set of solutions. While entropy regularization is a popular approach used to sustain exploration, it often proves unreliable for LLMs, suffering from high hyperparameter sensitivity and yielding only marginal performance gains. Motivated by these inefficiencies, we propose to rethink the relationship between policy entropy and exploration. By deriving a parametric formulation of group-relative advantage estimation and analyzing entropy dynamics, we conceptually decompose policy entropy into \textit{informative entropy}, which preserves diverse solution paths, and \textit{spurious entropy}, which erodes reasoning patterns. Our analysis reveals that, in contrast to blind maximization, effective exploration requires \textit{entropy refinement}-a mechanism implicitly embedded in group-relative advantage estimation that sustains informative entropy on positive rollouts while suppressing spurious entropy on negative ones. Guided by this insight, we propose \textbf{AsymGRPO}, an exploratory framework that explicitly decouples the modulation of positive and negative rollouts. This allows for independent control over the preservation of informative entropy and the suppression of spurious noise. Extensive experiments demonstrate that AsymGRPO achieves superior performance compared to strong baselines and exhibits the potential to synergize with existing entropy regularization methods.