An Agency-Transferring Model-Free Policy Enhancement Technique
2026-06-08 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors present a way to speed up reinforcement learning by starting with a working but imperfect policy and gradually shifting control to a new, trainable policy. This method ensures the agent reliably reaches its goal early in training by using the existing policy, then slowly lets the new policy take over completely. Their approach is backed by theoretical guarantees and shows better or comparable performance to other methods on control tasks, while consistently maintaining high success rates even after fully switching to the learned policy.
reinforcement learningbaseline policypolicy arbitrationcontinuous controlgoal-reachingneural networkstraining efficiencytheoretical analysispolicy transfer
Authors
Anton Bolychev, Georgiy Malaniya, Sinan Ibrahim, Pavel Osinenko
Abstract
Training reinforcement learning (RL) policies from scratch is costly: it requires careful reward and environment design, extensive tuning, and substantial computation. Yet many control problems already have a functional but suboptimal policy available as a baseline. This paper proposes a method for embedding such a baseline into the RL training process, simultaneously improving training efficiency relative to from-scratch methods and producing a learning policy that outperforms the baseline. At each step, the method arbitrates between the baseline policy and a trainable learning policy, initially relying strongly on the baseline policy and then progressively transferring agency to the learning policy. By the end of training, the learning policy is a standalone neural network that operates without baseline policy support. The paper formalizes what it means for the baseline policy to be functional: under this policy, the agent reaches a goal set and remains there with high probability. The proposed arbitration mechanism is designed to exploit this property during training, yielding high goal-reaching rates right from the beginning of training. A theoretical analysis provides a formal interpretation of this behavior under stated assumptions and extends it to the final baseline-free regime, where explicit lower bounds are derived for the goal-reaching probability of the standalone learning policy. Empirical results on continuous-control benchmarks show that the proposed method achieves returns that match or exceed those of competitive approaches, while maintaining the highest goal-reaching rates throughout training among the compared methods -- including in the final stage, where the learning policy operates without any baseline support.