From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents
2026-06-03 • Cryptography and Security
Cryptography and SecurityArtificial Intelligence
AI summaryⓘ
The authors explain that large language models (LLM) can perform complex tasks using tools, memory, and other aids, but this makes it hard to understand how they arrive at their answers. They focus on 'evidence tracing' and 'execution provenance,' which track how information and steps connect to produce final results. The authors review existing research, propose a framework to organize this work, and discuss challenges in making these AI processes more transparent and reliable. They also explore ways to better evaluate AI by looking beyond just whether the final answer is correct to how the AI got there.
Large Language ModelsEvidence TracingExecution ProvenanceAgent AutonomyTool UseMemory LineageDebuggingAuditObservabilityRuntime Guardrails
Authors
Yiqi Wang, Jiaqi Zhang, Taotao Cai, Zirui Liu, Qingqiang Sun, Zequn Sun, Zhangkai Wu, Mingkai Zhang, Yanming Zhu
Abstract
Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents. These capabilities expand agent autonomy, but also make agent behavior harder to verify, debug, and audit. Final-answer accuracy alone cannot explain how an output was produced, which evidence supported each claim, whether tool calls were justified, how memory influenced later decisions, or where execution failures originated. Evidence tracing and execution provenance address this gap by modeling how retrieved evidence, tool outputs, memory items, environment observations, intermediate claims, actions, and final answers are connected throughout agent execution. This survey provides a systematic review and conceptual framework for evidence tracing and execution provenance in LLM agents. We organize related work around a unified provenance perspective that connects retrieval grounding, claim support, tool-use safety, memory lineage, observability, debugging, audit, and recovery. We introduce a taxonomy covering trace sources, evidence and execution units, provenance relations, tracing granularity and timing, representation forms, and trust functions. We review key methodological directions, including provenance representation, evidence attribution, tool-use provenance, runtime guardrails, provenance-bearing memory, trace-based observability, and failure diagnosis. We also map existing benchmarks, datasets, and evaluation metrics to provenance-related capabilities, and discuss how evaluation can move from final-answer correctness toward process-level accountability. Finally, we outline open challenges, including unified trace schemas, claim-level and semantic provenance, provenance-aware safety mechanisms, realistic execution-trace benchmarks, recovery-oriented evaluation, and privacy-aware audit infrastructure.