Evaluation Dataset Versioning — Patterns

Identify a state by a digest of its contents

Problem. Human-applied tags get moved, reused and applied to the wrong branch. A result citing a tag therefore cites an intention rather than a fact, and when the tag is moved every historical result silently changes meaning with no record that anything happened.

Approach. Compute an identifier from the materialised items themselves — canonically serialised, ordered deterministically, digested — at the moment a run starts, and record it with the results. Human-readable names can point at digests for convenience, but the digest is what the result cites. Two runs agreeing on the digest saw identical data, regardless of what branch, tag or copy each was reading from.

Trade-off. A digest proves identity and explains nothing: when two differ, it cannot say what changed, so a readable diff is still needed alongside. Canonical serialisation is also fussy, and an incidental change to field ordering or whitespace handling produces a different digest for unchanged data, which erodes trust in the mechanism quickly.

Supersede items rather than editing expectations

Problem. An expectation turns out to be wrong and is corrected in place. Every historical verdict for that item was computed against the old expectation, and the item's pass history now mixes two different questions. Any trend analysis crossing that edit is wrong and nothing says so.

Approach. Treat a material change to an expectation as the creation of a new item that supersedes the old one, carrying a reference back to it. The old item is marked inactive from that version onward, and historical results stay attached to the expectation they were judged against. Trivial changes that cannot affect a verdict — a typo in a comment, an added label — are edits and are classified as such explicitly.

Trade-off. The collection accumulates inactive items that every query must filter, and the distinction between material and trivial is a judgement that will sometimes be made wrongly in the convenient direction. Supersession chains also become hard to read after several corrections to the same underlying case.

Classify every change so comparability is explicit

Problem. A version difference is reported as "the dataset changed", which forces the same investigation every time: did items get added, did expectations move, or was it only metadata. Those have completely different consequences for whether the two scores can be compared.

Approach. Produce a structured change record for every new version with counts in four categories: items added, items removed, expectations superseded, and metadata-only edits. Publish the record with the version. Comparison tooling reads it and states what the change permits — a metadata-only change preserves full comparability, additions preserve per-item comparability but not the aggregate, removals and supersessions break both unless the comparison is restricted to the stable intersection.

Trade-off. The classification has to be generated reliably, which means it is derived from a diff rather than declared by a person, and deriving material-versus-trivial automatically is imperfect. Restricting comparisons to the stable intersection is also the correct thing to do and makes the comparison population shrink quietly over time.

Pin referenced content, not references to it

Problem. Items that point at a document by location — a URL, a record identifier, a file path — are not versioned by versioning the item. The target changes, the pinned version no longer reproduces, and the failure appears as an unexplained score movement long after the change.

Approach. Materialise the referenced content into the version at the time the item is created or last reviewed, storing it by digest, and have the item cite that digest. Where the content is too large or cannot be copied for legal reasons, store a digest of it anyway and verify at run time, so that a divergence is reported as a reproducibility failure rather than scored as a quality change.

Trade-off. Materialising content multiplies storage and copies material whose retention and deletion obligations now apply to the evaluation collection too. Verification-only pinning avoids that and converts a silent corruption into a loud failure that blocks the suite, which is better but still stops work at an inconvenient moment.