Golden Datasets — References

Material on documenting what a dataset contains and why, on the quality of the labels inside it, and on what goes wrong when a reference set is built from what a system already does well.

Documenting a set

• Datasheets for Datasets — Timnit Gebru, Jamie Morgenstern, Briana Vecchione and colleagues, Communications of the ACM, 2021. The proposal that every dataset ship a document stating its motivation, composition, collection process and recommended uses, which is precisely the argument an evaluation set needs to carry and almost never does. Cited by name.
• Model Cards for Model Reporting — Margaret Mitchell, Simone Wu, Andrew Zaldivar and colleagues, Conference on Fairness, Accountability, and Transparency, 2019. Included here for the discipline of declaring the conditions under which an evaluation was performed, which is the reporting counterpart to documenting the set itself. Cited by name.

Label quality

• The literature on annotation disagreement in natural-language datasets — the finding that for many subjective tasks there is no single gold label and that forcing one discards real signal — is the strongest argument for assertion-based and rubric-based expectations. Described rather than attributed to a specific title, since the result appears across several independent lines of work.

Why a set from your own outputs cannot find your blind spots

• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (Thakur et al., 2021), arXiv:2104.08663 — cited here for the demonstration that performance on a set built in one domain transfers unevenly to another, which is the general form of the composition problem a curated set faces.
• Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020), arXiv:2009.03300 — useful as a worked example of deliberate subject-area composition, and of what a broad average does and does not reveal.

No verified reference is recorded for promotion of production traffic into curated evaluation items, for incident-seeded strata, or for the retention obligations that attach to customer content held in a test suite.