Benchmarks
Standard tests used to compare models on common tasks or datasets.
What does a two-point benchmark lead buy you?
Imagine a fixed suite of 100 catalogue questions. Model A answers 82 correctly and Model B answers 84. Both scores are meaningful only in relation to the same questions, scoring rules and allowed inputs.
- Protocol
- Record the dataset version, prompt, tools, model version and scoring procedure. A system given a search tool is being evaluated under a different condition from one answering from memory.
- Coverage
- The suite checks short factual questions. It has no questions requiring evidence from two competing catalogue editions.
- Comparison
- Inspect which items changed and whether the difference persists across relevant conditions. Two more correct answers alone do not establish a universal lead.
What is missing from the score?
A library deciding on a system may also care about citation accuracy, latency, cost and responses to unanswerable questions. Add a local test that resembles the actual workload and disclose where the benchmark and local task differ.

Could the model have seen the questions?
Public test items may appear in training or development. Document known overlap, use independent material where practical and describe uncertainty about exposure. A reproducible benchmark is a starting point for comparison, not a substitute for defining the job.
Follow the idea
Model Evaluation · Choose metrics and conditions for a real deployment.
Test Set · Check what makes the reported test independent.
Examples in the AI guides
Source · Stanford’s Holistic Evaluation of Language Models
Return to Benchmarks in the A–Z glossary