Prompt-Driven Exploration
2026-07-09 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors explain that in reinforcement learning (RL), trying new and different actions is important to improve, but simply adding random noise to actions doesn't help much. They propose using large language and vision-language models to change the policy by rewriting natural language prompts that guide the behavior. A vision-language model watches how the policy performs, figures out what went wrong, and adjusts the prompt to encourage better actions. This method, called Prompt-Driven Exploration (PDE), helps RL learn more effectively even when it starts with little or no rewards.
Reinforcement LearningExplorationLarge Language ModelsVision-Language ModelsPrompt EngineeringPolicyRolloutPosterior SamplingSample Efficiency
Authors
Sunshine Jiang, John Marangola, David Zhang, Raghuram Kowdeed, Ruiyang Luo, Nitish Dashora, Richard Li, Pulkit Agrawal, Zhang-Wei Hong
Abstract
Exploration is essential to RL since a policy cannot improve by repeatedly sampling the behaviors it already prefers. Standard methods inject stochasticity in the action space, but such jitter only yields rollouts close to the original. Escaping a weak policy often requires global perturbations that action noise cannot produce. Large language models (LLMs) and vision-language-action (VLA) models offer a pathway: they condition the policy on a natural language prompt, and since the rollout follows from it, modifying the prompt induces global changes. The challenge is finding prompts that induce useful global changes. With a weak policy that rarely succeeds, reward is too sparse to select on. Our idea is to refine prompts from the rollouts themselves: a vision-language model (VLM) reasons over the rollout video, diagnoses how the policy responded, and rewrites the prompt to elicit better behavior next time. This procedure realizes posterior sampling, a classical RL exploration framework, at the level of prompts: the VLM maintains an implicit distribution over useful prompts and updates it from observed rollouts. We call this strategy Prompt-Driven Exploration (PDE). Across manipulation and reasoning tasks, PDE enables RL to learn successful policies even from zero-reward starts, and improves sample efficiency more broadly. Our website is available at https://xinyunsunshine.github.io/prompt-rl.