A Variational Estimator for $L_p$ Calibration Errors
2026-02-27 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address the problem of calibration in machine learning, which means making sure predicted probabilities match actual outcomes. They improve on existing methods to estimate calibration errors, especially for multi-class predictions, by using a new approach that handles a wider variety of error types based on mathematical distances called L_p divergences. Their method can tell when predictions are too confident or not confident enough and avoids exaggerating errors. They tested their approach thoroughly and made their code publicly available.
calibrationmachine learningcalibration errormulticlass classificationvariational frameworkL_p divergencesprobability estimationoverconfidenceunderconfidenceproper losses
Authors
Eugène Berta, Sacha Braun, David Holzmüller, Francis Bach, Michael I. Jordan
Abstract
Calibration$\unicode{x2014}$the problem of ensuring that predicted probabilities align with observed class frequencies$\unicode{x2014}$is a basic desideratum for reliable prediction with machine learning systems. Calibration error is traditionally assessed via a divergence function, using the expected divergence between predictions and empirical frequencies. Accurately estimating this quantity is challenging, especially in the multiclass setting. Here, we show how to extend a recent variational framework for estimating calibration errors beyond divergences induced induced by proper losses, to cover a broad class of calibration errors induced by $L_p$ divergences. Our method can separate over- and under-confidence and, unlike non-variational approaches, avoids overestimation. We provide extensive experiments and integrate our code in the open-source package probmetrics (https://github.com/dholzmueller/probmetrics) for evaluating calibration errors.