Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning
2026-04-10 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study how confidence scores, which show how sure a language model is about its answers, change after extra training called supervised fine-tuning (SFT). They find that after SFT, these scores become less reliable because they can be influenced by things like how similar the output is to the training data instead of just quality. This makes the confidence scores less helpful, especially in real tasks. Their work suggests that confidence measures need to be carefully checked and improved when models are fine-tuned.
Uncertainty quantificationLanguage modelsConfidence scoresSupervised fine-tuning (SFT)Model calibrationOutput qualityHallucination detectionTraining distributionCorrelation analysisDownstream tasks
Authors
Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Jackie Chi Kit Cheung
Abstract
Uncertainty quantification is a set of techniques that measure confidence in language models. They can be used, for example, to detect hallucinations or alert users to review uncertain predictions. To be useful, these confidence scores must be correlated with the quality of the output. However, recent work found that fine-tuning can affect the correlation between confidence scores and quality. Hence, we investigate the underlying behavior of confidence scores to understand its sensitivity to supervised fine-tuning (SFT). We find that post-SFT, the correlation of various confidence scores degrades, which can stem from changes in confidence scores due to factors other than the output quality, such as the output's similarity to the training distribution. We demonstrate via a case study how failing to address this miscorrelation reduces the usefulness of the confidence scores on a downstream task. Our findings show how confidence metrics cannot be used off-the-shelf without testing, and motivate the need for developing metrics which are more robust to fine-tuning.