Target Policy Optimization
2026-04-07 • Machine Learning
Machine Learning
AI summaryⓘ
The authors discuss a new method called Target Policy Optimization (TPO) to improve reinforcement learning. Instead of combining how to adjust probabilities and parameters into one step like usual, TPO separates them. It first creates a target distribution based on the old policy and scores, then fits the model to this target, making updates more stable. They show that TPO works as well as or better than common methods, especially when rewards are rare or sparse.
Reinforcement LearningPolicy GradientTarget Policy OptimizationCross-EntropySparse RewardTransformerLarge Language ModelsPolicy UpdateBandit ProblemLogits
Authors
Jean Kaddour
Abstract
In RL, given a prompt, we sample a group of completions from a model and score them. Two questions follow: which completions should gain probability mass, and how should the parameters move to realize that change? Standard policy-gradient methods answer both at once, so the update can overshoot or undershoot depending on the learning rate, clipping, and other optimizer choices. We introduce \emph{Target Policy Optimization} (TPO), which separates the two questions. Given scored completions, TPO constructs a target distribution $q_i \propto p_i^{\,\mathrm{old}} \exp(u_i)$ and fits the policy to it by cross-entropy. The loss gradient on sampled-completion logits is $p^θ- q$, which vanishes once the policy matches the target. On tabular bandits, transformer sequence tasks, and billion-parameter LLM RLVR, TPO matches PG, PPO, GRPO, and DG on easy tasks and substantially outperforms them under sparse reward. Code is available at https://github.com/JeanKaddour/tpo.