Overfitting

Overfitting happens when a model learns quirks of its training examples that carry over poorly to new cases.

Which tree would you take to the next orchard?

Imagine two decision trees trained to identify blemished apples. Both see the same training photos and are compared on the same separate validation set.

Shallow tree

Training accuracy is 90%. Validation accuracy is 88%.

Deeper tree

Training accuracy rises to 99%. Validation accuracy falls to 82%.

The deeper tree fits the training photos more closely. The shallow tree does better on the evidence that matters for choosing between them here.

Investigate the gap

One possibility is that the deeper tree learned marks left by a particular sorting tray. Inspect errors from different trays and orchard batches. Keep a final independent test set for the model selected on validation results. Repeated adjustments to the validation set can make its score optimistic too.

Follow the idea

Training set explains which observations influence the fit.

Model drift covers a different reason performance may change after deployment.

Source · scikit-learn’s guide to cross-validation

Return to Overfitting in the A–Z glossary