Cost-Sensitive Conformal Prediction and Human-in-the-Loop Abstention for Imbalanced High-Stakes Decision Support: A Multi-Domain Benchmark
2026-07-29 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied how current methods to measure uncertainty in important decision systems, like credit scoring and healthcare, often fail to properly account for rare but costly mistakes. They found that a common technique called marginal conformal prediction severely underestimates uncertainty for minority classes, leading to poor coverage. Their experiments showed that another method, Mondrian conformal prediction, greatly improves coverage for these rare classes. Combining this with a strategy to defer uncertain cases to human experts also reduces overall costs, offering practical advice for safer and fairer decision systems.
conformal predictionclass imbalanceuncertainty quantificationminority class coveragecost-controlled abstentionMondrian conformal predictionhigh-stakes decision systemsprobability calibrationhuman-in-the-loopdecision cost
Authors
Manpreet Singh, Akshatha Srikantha, Shyamal Lakhanpal
Abstract
High-stakes decision systems in credit scoring, fraud detection, healthcare, and industrial safety require reliable uncertainty quantification under severe class imbalance and asymmetric error costs. Standard marginal conformal prediction (CP) provides valid overall coverage guarantees; however, we show that it severely under-covers rare, costly minority classes, with minority-class coverage dropping to as low as 0.5% on certain datasets. To characterize and address this limitation, we conduct a comprehensive benchmark comparing marginal CP, class-conditional (Mondrian) CP, and cost-controlled abstention mechanisms across 15 real-world imbalanced tabular datasets, 7 classification models, 3 probability calibration techniques, and 10 random seeds, resulting in 3,150 experimental runs. Our results show that Mondrian CP restores valid minority-class coverage, achieving an average minority-coverage improvement of 61.7 percentage points over marginal CP (p < 1e-80). Furthermore, combining Mondrian CP with cost-controlled abstention significantly reduces expected decision cost compared with standard decision boundaries, confidence-based rejectors, and risk-controlled rejectors under realistic human review budgets. We further quantify dataset-specific break-even thresholds at which deferring ambiguous instances to human experts becomes cost-effective. These findings provide practical guidance for deploying distribution-free, cost-aware uncertainty quantification in high-stakes decision support systems.