Evaluation Dataset Versioning — References

Material on documenting what a dataset state contains, on content-addressed identity, and on the reproducibility claims a pinned version is supposed to support.

Documenting a state

• Datasheets for Datasets — Timnit Gebru, Jamie Morgenstern, Briana Vecchione and colleagues, Communications of the ACM, 2021. Relevant here for the insistence that a dataset carries a record of its provenance and its changes, which is what a version annotation has to supply for each successive state. Cited by name.

Content-addressed identity

• The content-addressable storage model, in which an object's identifier is a cryptographic digest of its bytes so that identity and integrity are the same property, is the mechanism behind digest-based dataset versioning. Named as a model; it is implemented in widely used version control and storage systems rather than described in one canonical paper.

Reproducibility

• The machine-learning reproducibility literature — which consistently finds that the data state, not only the code and the seeds, determines whether a result can be reproduced — is the standing argument for recording a version identifier on every stored score. Described as a body of work rather than attributed to a single title.
• Model Cards for Model Reporting — Margaret Mitchell, Simone Wu, Andrew Zaldivar and colleagues, Conference on Fairness, Accountability, and Transparency, 2019. Included for the requirement that a reported result declare the evaluation data it was computed over. Cited by name.

No verified reference is recorded for supersession of evaluation items over in-place correction, for change classification as a comparability contract, or for content pinning of externally referenced material in evaluation sets.