Trace-Mediated Peak Bias: Bridging Temporal Credit Assignment and Cognitive Heuristics in Deep Reinforcement Learning
2026-06-03 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied how learning over time works in both brains and AI, focusing on a problem they call Trace-Mediated Peak Bias (TMPB). They found that when AI agents look back at past rewards, they sometimes wrongly focus too much on big reward 'peaks' instead of the total reward, similar to how humans remember events by their most intense moments (the Peak-End Rule). This happens because certain learning methods can't properly adjust very large error signals, causing the AI to overestimate some actions. However, the authors show that using adaptive learning methods can fix this problem. Their work suggests that some human memory biases might come from basic mathematical limits in learning systems.
Temporal credit assignmentEligibility tracesDeep reinforcement learningTemporal Difference errorsStochastic Gradient DescentAdaptive optimizationPeak-End RuleGradient shocksValue estimationSaliency distortions
Authors
Viktor Veselý, Aleksandar Todorov, Erwan Escudie, Matthia Sabatelli
Abstract
Temporal credit assignment is central to both biological and artificial intelligence, yet its interaction with non-linear function approximation is poorly understood. We identify a systematic failure mode in deep reinforcement learning (RL) termed Trace-Mediated Peak Bias (TMPB). At intermediate eligibility trace depths, agents irrationally prefer trajectories with high-magnitude reward ``peaks'' over alternatives with higher cumulative returns. This provides a mechanistic account of the Peak-End Rule: a human memory bias where experiences are judged by their most intense moments rather than integrated utility. We show that TMPB emerges because traces amplify distal Temporal Difference errors into ``gradient shocks'' that fixed-step-size Stochastic Gradient Descent cannot normalize, leading to global overestimation. Conversely, adaptive optimizers mitigate this pathology via second-moment normalization. Our results suggest that human-like saliency distortions may emerge naturally from the mathematical constraints of credit assignment in distributed systems, and that adaptive optimization is a theoretical necessity for rational value estimation.