Adaptive Fault Injection Planning for Multi-Layer Self-Healing AI Infrastructure
2026-07-17 • Emerging Technologies
Emerging Technologies
AI summaryⓘ
The authors studied how faults (errors) spread across different parts of GPU systems, from hardware to software, and found that many traditional tests miss these cross-layer problems. They created ADA-ST, a method that smartly chooses fault scenarios to test all possible fault paths between layers. Using real data from a large GPU platform, ADA-ST tested fault spread much more thoroughly and adapted the testing for newer platforms as well. Their work revealed hidden weaknesses in detection methods and communication between layers in the GPU stack.
GPU acceleratorfault injectioncross-layer fault propagationself-healing pipelinefault diagnosisfault-injection campaigntelemetrymulti-signal correlationL2-L3 cache boundaryfault coverage
Authors
Saurabh Kulkarni, Yuxin Yang, Rohan Kulkarni, Gautam Nayak
Abstract
Modern GPU-accelerator platforms rely on multi-layer self-healing pipelines that span hardware, firmware, management software, and orchestration. When faults propagate across layer boundaries, they can bypass detection, corrupt diagnosis, or trigger conflicting remediations--yet conventional fault-injection campaigns test each layer in isolation. We present ADA-ST, an adaptive fault-injection methodology that uses a weighted fault-propagation graph to guide cross-layer scenario selection. We construct four-layer graphs for three successive platforms at a hyperscale operator: Alpha, Beta, and Gamma. Platform Alpha, a production system that accumulated 72,550 repair tickets over four years, provides the empirical foundation; 49% of those tickets involve cross-layer fault propagation. We show that existing static test campaigns cover only 20-25% of the modeled fault-propagation edges, leaving approximately three-quarters of the cross-layer attack surface unexercised. ADA-ST closes this gap through iterative, activity-guided scenario selection that maximizes marginal coverage gain per iteration, reaching full edge coverage within 10 iterations on Alpha, 12 on Beta, and 9 on Gamma. The Fault-Layer Abstraction Mapping (FLAM) transfers propagation knowledge across hardware generations with 100% fidelity from Alpha to Beta and 96% from Beta to Gamma. Physical spot-validation on the newest platform confirms all four tested propagation edges, revealing cross-layer vulnerabilities spanning telemetry blind spots, absence-based detection gaps, multi-signal correlation failures, and trust-without-verification propagation at the L2-to-L3 boundary.