Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
2026-06-02 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors found that the popular reinforcement learning method PPO struggles to adapt when the environment changes over time because it makes small, inefficient updates without considering the underlying geometry of policy changes. They introduce a new method called Gaussian Trust Region Policy Optimization (GTR) that uses a Gaussian-based constraint to allow more flexible and stable updates, especially when big changes are needed. They also add a technique to reduce noise from old policy references. Their approach works well across different tasks and types of environments, showing that understanding policy geometry can help in complex, changing situations.
Proximal Policy Optimization (PPO)reinforcement learningnon-stationary environmentstrust region methodspolicy optimizationGaussian kernelregularizationpolicy adaptationlocal updatesgeometry-aware optimization
Authors
Bingxu Liu, Jiashun Liu, Johan Obando-Ceron, Hao Wang, Runze Liu, Pablo Samuel Castro, Aaron Courville, Ling Pan
Abstract
While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments. The failure does not stem from insufficient model capacity or overly restrictive clipping. Instead, PPO performs persistent, directionally inefficient local updates, which indicates a lack of geometry-aware guidance for accumulating meaningful behavioral change and ultimately hindering transitions toward new behavior patterns. Although divergence-based regularization introduces partial geometric awareness, its monotonically increasing penalties implicitly discourage large policy deviations, even when such shifts are necessary for effective adaptation. To address this limitation, we propose Gaussian Trust Region Policy Optimization (GTR), which reshapes the trust region using a Gaussian kernel. The resulting constraint is bounded and non-monotonic, providing strong local stability while progressively relaxing under sustained high-advantage updates. To further improve robustness, we introduce a Mixture Gaussian Anchor that adapts to recent policy trajectories, reducing variance induced by stale references. GTR is architecture-agnostic and achieves strong performance across games, simulated robotic control, open-world exploration, and language model post-training. These results demonstrate that geometry-aware trust-region design can be a promising direction for robust reinforcement learning in complex non-stationary environments. Our code is available at https://anonymous.4open.science/r/GTR_demo/README.md.