Benchmarks

Standard tests used to compare models on common tasks or datasets.

What does a two-point benchmark lead buy you?

Imagine a fixed suite of 100 catalogue questions. Model A answers 82 correctly and Model B answers 84. Both scores are meaningful only in relation to the same questions, scoring rules and allowed inputs.

Protocol
Record the dataset version, prompt, tools, model version and scoring procedure. A system given a search tool is being evaluated under a different condition from one answering from memory.
Coverage
The suite checks short factual questions. It has no questions requiring evidence from two competing catalogue editions.
Comparison
Inspect which items changed and whether the difference persists across relevant conditions. Two more correct answers alone do not establish a universal lead.

What is missing from the score?

A library deciding on a system may also care about citation accuracy, latency, cost and responses to unanswerable questions. Add a local test that resembles the actual workload and disclose where the benchmark and local task differ.

Two stopwatches sit above matched botanical cards.
Comparable evaluations require both the same task and the same measurement conditions.

Could the model have seen the questions?

Public test items may appear in training or development. Document known overlap, use independent material where practical and describe uncertainty about exposure. A reproducible benchmark is a starting point for comparison, not a substitute for defining the job.

Follow the idea

Model Evaluation · Choose metrics and conditions for a real deployment.

Test Set · Check what makes the reported test independent.

Examples in the AI guides

Source · Stanford’s Holistic Evaluation of Language Models

Return to Benchmarks in the A–Z glossary