To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning
2026-07-12 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors found that current training methods for language models with search tools mostly reward correct answers but don't punish wrong made-up answers, which makes the models hallucinate more. To fix this, they created a new training method called Abstention-Aware Reinforcement Learning (AWA-RL) that teaches the model when it's better to say "I don't know" based on how confident it is. They also designed a new measurement (RA-F1) to check how well models balance being accurate and reliable. Their method improved the model's precision and reliability without losing much accuracy, making the models better at knowing when to answer or abstain.
Large Language ModelsReinforcement LearningOpen-Domain Question AnsweringHallucinationAbstentionPrecisionOn-Policy TrainingEvaluation MetricsSearch ToolsReliability
Authors
Fengji Zhang, Tianyu Fan, Yuxiang Zheng, Xinyao Niu, Chengen Huang, Jacky Keung, Bei Chen
Abstract
Recent advances in equipping Large Language Models (LLMs) with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's query-specific prior capabilities and continuous on-policy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at https://github.com/zfj1998/AWA-RL.