Calibration

The degree to which a model's confidence matches what really happens.

Two shelves of rain forecasts

Imagine a forecaster issues probabilities of measurable rain for the next day. Group past forecasts by their stated probability, then count rainy days in each group.

20% forecasts
10 / 50
Out of 50 days given a 20% chance, it rained on 10. The observed frequency is 10 ÷ 50 = 20%.

80% forecasts
30 / 50
Out of 50 days given an 80% chance, it rained on 30. The observed frequency is 30 ÷ 50 = 60%.

The second group was overconfident in rain by 20 percentage points in this sample.

What a reliability diagram shows

Plot each group’s mean predicted probability against the fraction of events observed. Points near the diagonal show agreement. Small groups can move substantially by chance, so report how many forecasts went into each point. A separate evaluation set helps assess how calibration holds up on new cases.

Follow the idea

Confidence describes an individual prediction’s stated certainty.

Model monitoring tracks whether the observed agreement changes over time.

Examples in the AI guides

Related Yenra articles (5)

Source · scikit-learn’s probability calibration guide

Return to Calibration in the A–Z glossary