Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents
2026-08-18 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study how large language models (LLMs) can be combined with reinforcement learning (RL) by modeling the system as a special kind of decision problem called a Goal-Augmented Markov Decision Process. They show that using the LLM's score as a bounded shaping function does not change the best actions to take, even if the LLM scores are not perfect. This result is stronger than previous methods that directly use LLM outputs as rewards. They also tested their theory with simulations, including difficult cases where the LLM scores were scaled up a lot.
large language modelsreinforcement learningMarkov Decision Processpotential-based reward shapingoptimal policyLLM reward signalsGoal-Augmented MDPpolicy invariancenumerical verification
Authors
Christophe D. Hounwanou, John Emeka Eze, Yaé U. Gaba
Abstract
Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.