Evaluation Dataset Versioning

What it is

Giving every state of an evaluation collection a name, so that a recorded score can be traced back to exactly the items and expectations it was computed over. It covers how a state is identified, how two states are compared, what happens to historical results when an item's expectation is corrected, and how a run from eighteen months ago can be reproduced. It takes no position on what belongs in the collection; its question is which collection a number came from, and whether two numbers were computed over the same one.

Why it matters

Evaluation collections change constantly — items added after an incident, expectations corrected, near-duplicates removed, a whole category imported. Every one of those edits changes what a score means, and none of them is visible in the score. Without versioning, a quality trend is a sequence of measurements over quietly different populations, and the most common way a team discovers this is by spending a week investigating a regression that was somebody adding forty hard items. The second reason is evidential: where a release decision or a regulatory submission rests on an evaluation result, being unable to say what that result was computed over makes the result unusable as evidence.

Key decisions

• What identifies a state: a tag applied by a person, a content digest computed over the items, or a commit in a repository. Only a digest cannot be applied to the wrong thing.
• Whether versions are immutable once referenced by a result, or whether in-place correction is permitted and recorded.
• How a change is classified — items added, items removed, expectations corrected, metadata only — since only the last leaves historical comparability intact.
• Whether a correction to an expectation creates a new item that supersedes the old one, or edits the existing item, which decides whether historical per-item verdicts remain interpretable.
• How large the collection is allowed to get before full-copy storage stops being viable, and whether items are stored by content or by reference to a source that can itself change.
• How frequently a suite advances to the newest version, and whether a release comparison is permitted to span two versions with an explicit acknowledgement.
• Whether every stored result carries the version identifier as a required field, since a scheme that is optional on the result side provides no guarantees at all.

Failure modes

• Scores are stored with a date and no version, and nothing distinguishes a quality movement from a collection that grew by ten percent that week.
• A version tag is moved to point at a later state, and every result referencing it silently changes meaning.
• A wrong expectation is corrected in place, historical pass rates for that item become nonsensical, and a year of trend data for the affected category is quietly invalid.
• Items reference external documents by URL rather than by stored content, the documents change, and a pinned version no longer reproduces anything.
• Candidate and baseline are run against different versions because one branch had newer items, and the resulting difference is reported as a model regression.
• The collection is versioned and the scorers are not, so two runs against the same pinned version are still incomparable.
• Every edit creates a version, there are four hundred of them with no annotation, and finding the one a result referred to is archaeology.
• Removing an item to fix a duplicate changes the aggregate, and because removals are not reported separately from score changes, nobody can tell which happened.
• The version lives in the evaluation tooling and the results live in a warehouse, and the join between them was never built.

Related capabilities

• Golden Datasets — the curation this capability names successive states of
• Regression Evaluation — the consumer that must establish both runs saw the same items
• Offline Evaluation — records the version on every run it produces
• Evaluation Gates — a threshold is only meaningful against a stated version
• Safety Evaluation — where reproducibility of a past result is an evidential requirement

Applies: Production readiness · Design questions