Calibration
The degree to which a model's confidence matches what really happens.
Two shelves of rain forecasts
Imagine a forecaster issues probabilities of measurable rain for the next day. Group past forecasts by their stated probability, then count rainy days in each group.
20% forecasts
Out of 50 days given a 20% chance, it rained on 10. The observed frequency is 10 ÷ 50 = 20%.
80% forecasts
Out of 50 days given an 80% chance, it rained on 30. The observed frequency is 30 ÷ 50 = 60%.
The second group was overconfident in rain by 20 percentage points in this sample.
What a reliability diagram shows
Plot each group’s mean predicted probability against the fraction of events observed. Points near the diagonal show agreement. Small groups can move substantially by chance, so report how many forecasts went into each point. A separate evaluation set helps assess how calibration holds up on new cases.
Follow the idea
Confidence describes an individual prediction’s stated certainty.
Model monitoring tracks whether the observed agreement changes over time.
Related concepts
Examples in the AI guides
Related Yenra articles (5)
Source · scikit-learn’s probability calibration guide
Return to Calibration in the A–Z glossary