Cross-Validation

A method of testing a model by training and validating it across multiple different splits of the data.

How do three folds produce one development estimate?

A photo archive divides 30 distinct objects into groups A, B and C, ten objects each. Each object has several scans. For this example a model is scored once per object, and every scan of an object stays in its group.

Training groupsValidation groupCorrect of ten
B and CA8
A and CB9
A and BC7

What are you averaging?

Fit a fresh model in each row. The three validation results total 24 correct out of 30 out-of-fold predictions, or 80%. The spread from 70% to 90% is worth reporting. These are overlapping training sets, so the three scores are not independent experiments.

Three separate folders hold groups of photographs.
Each group takes one turn in validation. Views of the same object belong together.

Where do preprocessing and tuning belong?

Fit learned preprocessing only on the training portion of each fold. If the same folds guide a search for the best model, use a separate test set or nested evaluation for a final performance estimate. For future-time prediction, use splits that respect time order.

Follow the idea

Validation Set · Identify which choices use development data.

Test Set · Distinguish resampling from independent final testing.

Examples in the AI guides

Source · Scikit-learn cross-validation guide

Return to Cross-Validation in the A–Z glossary