AI summaryⓘ
The authors studied how automated systems find the root cause of problems in microservices, pointing out that simply identifying the faulty service isn’t enough. They created a new way to evaluate how these systems investigate faults step-by-step by comparing their reasoning paths against expert-designed fault routes. Their findings showed that a system might guess the right faulty service but still miss important details about how the fault spread, which is crucial for fixing issues. To improve this, they developed DiagGuard, a two-part system that first checks what evidence is available and then verifies the diagnosis, which improved accuracy in tests. This work helps reveal hidden weaknesses in current evaluation methods and suggests ways to make root cause analysis more reliable.
root cause analysismicroservicesfault propagationdiagnostic trajectorysite reliability engineertelemetrylocalization accuracyDiagGuardautomated diagnosisevaluation metrics
Authors
Qisheng Lu, Aoyang Fang, Junjielong Xu, Jin'ao Shang, Songhan Zhang, Yifan Yang, Xiaochuan Yan, Pinjia He
Abstract
Existing evaluations of automated root cause analysis (RCA) for microservices assess diagnostic performance mainly by endpoint correctness: whether a method localizes the responsible service. This criterion enables comparison but does not reveal the evidentiary basis of a diagnosis or the fault-propagation route connecting the source to observed symptoms, both of which an on-call site reliability engineer needs to judge whether action is warranted. We therefore treat RCA as an observable diagnostic process. Our trajectory-level framework evaluates agent executions against manually curated service-level fault-propagation paths. Applied to a public microservice RCA benchmark, it analyzes 3,500 diagnostic trajectories, characterizing where agents investigate and how they use retrieved telemetry. We find a disconnect between answer correctness and diagnostic quality: an agent may localize the fault source yet fail to reconstruct its propagation. Successful investigations stay on the fault-impact surface, act on retrieved evidence, and broaden their query repertoire as the search deepens. Failures arise when decisive evidence is omitted, retrieved evidence is misinterpreted, or unsupported inference substitutes for missing evidence. We operationalize this taxonomy as DiagGuard, a two-stage defense-in-depth architecture in which grounding surveys available observations before localization and verification audits the diagnosis against them. In an independent setting with a different model, benchmark, and service topology, DiagGuard raises Acc@1 from 43.5% to 52.5%. These results show that trajectory-level evaluation exposes limitations hidden by final-answer metrics and provides actionable guidance for improving automated RCA.