Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling
2026-04-09 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors noticed that current reward models used to teach AI through human feedback struggle when the AI acts like an agent using tools and making complex plans. To help fix this, they created Plan-RewardBench, a new test that checks how well models can tell which AI action plans are better in tricky tool-related tasks. They tested different types of reward models and found all of them have trouble, especially with longer and more complicated tasks. Their work highlights the need for better training methods for these models when dealing with agent-like AI behaviors.
Reinforcement Learning from Human FeedbackReward ModelsLarge Language ModelsAgentic SystemsTool IntegrationTrajectoryBenchmarkPreference ModelingComplex PlanningError Recovery
Authors
Jiaxuan Wang, Yulan Hu, Wenjin Yang, Zheng Pan, Xin Li, Lan-Zhe Guo
Abstract
In classical Reinforcement Learning from Human Feedback (RLHF), Reward Models (RMs) serve as the fundamental signal provider for model alignment. As Large Language Models evolve into agentic systems capable of autonomous tool invocation and complex reasoning, the paradigm of reward modeling faces unprecedented challenges--most notably, the lack of benchmarks specifically designed to assess RM capabilities within tool-integrated environments. To address this gap, we present Plan-RewardBench, a trajectory-level preference benchmark designed to evaluate how well judges distinguish preferred versus distractor agent trajectories in complex tool-using scenarios. Plan-RewardBench covers four representative task families -- (i) Safety Refusal, (ii) Tool-Irrelevance / Unavailability, (iii) Complex Planning, and (iv) Robust Error Recovery -- comprising validated positive trajectories and confusable hard negatives constructed via multi-model natural rollouts, rule-based perturbations, and minimal-edit LLM perturbations. We benchmark representative RMs (generative, discriminative, and LLM-as-Judge) under a unified pairwise protocol, reporting accuracy trends across varying trajectory lengths and task categories. Furthermore, we provide diagnostic analyses of prevalent failure modes. Our results reveal that all three evaluator families face substantial challenges, with performance degrading sharply on long-horizon trajectories, underscoring the necessity for specialized training in agentic, trajectory-level reward modeling. Ultimately, Plan-RewardBench aims to serve as both a practical evaluation suite and a reusable blueprint for constructing agentic planning preference data.