Offline Evaluation — References

Material on why a fixed measurement set decays, what a held-out split is actually protecting, and how much of a reported difference is noise.

Held-out data and what breaks it

• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (Dodge et al., 2021), arXiv:2104.08758 — benchmark contamination found in a widely used pretraining corpus, which is the mechanism by which a held-out set can be compromised before anyone on your team touches it.
• The literature on adaptive data analysis — repeatedly querying a held-out set and making decisions on the answers until the set no longer generalises — is the formal statement of why a reserved slice degrades through aggregate scores alone. Described rather than cited to a specific title, because several distinct papers in that line are routinely conflated.

Reporting a difference honestly

• The practice of reporting variance across multiple runs rather than a single score, widely argued for in machine-learning reproducibility work, is the only defence against a suite whose run-to-run spread exceeds the effects it is used to adjudicate. No single canonical reference is recorded here.

Multi-metric reporting

• Holistic Evaluation of Language Models (Liang et al., 2022), arXiv:2211.09110 — the argument that a single quality number is an inadequate basis for a decision, and a worked example of reporting several metrics against one fixed set of scenarios.

No verified reference is recorded for dependency record-and-replay as an evaluation harness technique, for run-configuration provenance, or for the economics of suite runtime against change frequency. These are engineering practices carried by tooling documentation rather than by published measurement.