Calibration is the degree to which a model's predicted confidence matches what actually happens in the real world. If a model says "80% probability" across many similar cases, then roughly 80% of those cases should truly be positive for the model to be well calibrated. Calibration is about honesty in probabilities, not just whether the top answer is correct.
Why Calibration Matters
A model can have good accuracy and still be badly calibrated. That becomes a serious problem when people use the model's confidence to decide whether to trust a prediction, escalate to a human, or take an expensive action. In medicine, finance, security, and content moderation, overconfident predictions can be as dangerous as inaccurate ones.
This is why calibration is closely tied to Model Evaluation. Teams often need more than raw accuracy. They need to know whether probability estimates are dependable enough for decision-making.
How Teams Measure It
Calibration is often examined with reliability diagrams and calibration curves, which compare predicted probabilities to observed outcomes. Teams may also use summary metrics such as Expected Calibration Error. If the model is miscalibrated, they may apply techniques such as temperature scaling or other post-processing methods to bring confidence scores closer to reality.
The underlying challenge is that training for accuracy does not automatically train for trustworthy confidence. Models can become overconfident on familiar patterns, underconfident on borderline cases, or unstable when the data changes after deployment.
Calibration in Modern AI
Calibration matters beyond classic classifiers. Search ranking, recommendation systems, anomaly detection pipelines, and generative AI safety systems all rely on confidence signals of some kind. Even when a language model does not output a neat probability to the user, teams still need internal signals that help decide when to trust the system and when to add safeguards.
Good calibration does not solve every trust problem, but it makes the system easier to govern. It gives humans a more reliable sense of uncertainty and helps match automation to the real level of risk.
Related concepts: Confidence, Uncertainty, Model Evaluation, Precision, Recall, and Model Monitoring.