Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games
2026-07-12 • Multiagent Systems
Multiagent SystemsArtificial Intelligence
AI summaryⓘ
The authors study how to evaluate AI agents playing a complex hidden-role game called Werewolf, where it's hard to tell why agents make certain decisions. They created a framework that tracks what the agents believe about hidden roles and logs how their beliefs change and influence actions, helping identify mistakes. Their experiments showed that agents using active belief tracking helped the 'good' team win more often, but they couldn't explain exactly why this happened, as belief and action were only weakly connected. The main contribution is the auditing framework, which makes agent decisions more transparent and supports safer improvement cycles.
LLM agentshidden-information gamesWerewolf gamebelief trackingauditing frameworkmulti-agent systemsaction-belief consistencyoffline improvementstrategy evaluationhigh-noise environments
Authors
Yuan Gao, Jiangyi Yang, Yao Zhao, Yichi Zhang
Abstract
Evaluating LLM agents in hidden-information multi-agent settings is hard: final outcomes are high-variance and rarely reveal why an agent decided as it did. We study this in a 9-player Werewolf environment where agents act under strict, code-level information isolation, and we build an auditable framework that maintains an external belief state over hidden roles, logs belief updates and belief-action deviations as structured evidence, and supports a defensive offline improvement loop that reviews bad cases before any strategy change. Across 1,080 frozen games spanning belief-disabled, active-belief, kernel-ablation, camp-restricted, consumption-policy, and high-load arms, and including a seed-paired A0/A1 comparison, the active-belief condition is associated with substantially better good-side outcomes: in the 200-seed A0/A1 comparison the good-side win rate rises from 0.205 to 0.390 (paired McNemar $χ^2 = 16.4$, $p < 0.001$), with fewer irreversible witch-poison errors. We do not, however, attribute this shift to belief content. Direct action-belief consistency is low ($\approx 0.21$), and giving belief only to the werewolves helps the good side more than giving it only to the good side, which argues against a simple holder-benefit account; we therefore report the effect as an association and treat its mechanism as unresolved. The contribution is the audit framework itself: it makes the effect measurable, exposes low direct action-belief consistency, rejects an unreliable forced-consumption intervention with evidence, and separates strategy effects from load confounds. We accordingly position external belief in high-noise hidden-information games primarily as an auditable cognitive baseline that also carries decision-relevant signal, turning opaque agent behavior into replayable evidence for safer, controlled iteration.