CRRL: A Causality-Based Reinforcement Learning Framework for Autonomous System Recovery
2026-07-03 • Software Engineering
Software EngineeringArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors point out that usual reinforcement learning methods struggle to recover well from problems in driving tasks because they don’t understand cause-and-effect and often get stuck. They propose a new approach called CRRL that teaches driving policies to work better with simple rule-based recovery helpers by using causal knowledge from real driving data. This method helps the policies predict trouble and act accordingly, improving overall performance and reducing the need for recovery. Tests in different driving scenarios showed that CRRL leads to better rewards and smoother driving with fewer interventions.
Reinforcement LearningCausal InferenceAutonomous SystemsPolicy TrainingRule-based RecoveryPPO (Proximal Policy Optimization)Failure RecoveryDriving ScenariosMAPE-K Framework
Authors
Safia Fatima, Kai Olav Ellefsen, Leon Moonen
Abstract
Traditional reinforcement learning (RL) for recovery in autonomous systems lacks causal understanding and generalizes poorly to novel failure scenarios. RL policies often stall in failure states, spending up to 70% of an episode immobilized. Rule-based recovery alone is inadequate, and adding heuristic recovery to a pretrained PPO policy worsens rewards because policies cannot coordinate well with unanticipated interventions. The issue is not missing recovery mechanisms but a lack of policies trained to collaborate with them. We introduce CRRL, a causal-guided RL framework that trains policies to work effectively with rule-based recovery. The recovery detects stalled states and assists the agent. Causal relations from driving logs shape the training signal, teaching the policy to anticipate stalls and adjust actions in recovery contexts. The framework follows MAPE-K, with sensor collection, causal model construction, and hybrid RL policy training corresponding to Monitor, Analyze, and Plan/Execute, respectively. We evaluate CRRL through a four-condition ablation study across three driving scenarios, with 20 episodes per condition. We find that causal training significantly improves reward, distance, and velocity. Moreover, 9 of 20 roundabout episodes required zero recovery intervention, confirming navigation competence. These results show that causal-guided training produces effective RL policies that cooperate with rule-based safety components.