Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
2026-07-11 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors explain that a common method used to train language models with reinforcement learning, called PPO-Clip, has a hidden problem because it measures policy changes in a way that doesn’t match the true shape of the policy space. This mismatch causes the model to either explore too little in some areas or make big jumps in others, leading to poor learning. To fix this, they introduce Riemannian Isometric Policy Optimization (RIPO), which respects the true geometric nature of policies and keeps updates balanced. Their experiments show that RIPO improves performance on several challenging benchmarks compared to earlier methods.
Reinforcement LearningPPO-ClipPolicy OptimizationRiemannian ManifoldPolicy GeometryExploration-ExploitationBias-Variance Trade-offLanguage ModelsPolicy UpdateGRPO
Authors
Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Mingxuan Wang, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
Abstract
Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. However, RL algorithms with PPO-Clip are inherently limited by exploration collapse. Subsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip's failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable bias-variance trade-off, which stabilizes optimization. Extensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24).