Golden Datasets

What it is

The curated collection of inputs, paired with a statement of what the system ought to do with each, that every pre-release measurement is taken against. Its content is an argument: these cases, in these proportions, are what this product is for. Building one is mostly editorial work — deciding what belongs, deciding who is authoritative about the right answer, and writing that answer down in a form a scorer can use without re-interpreting it.

Why it matters

A reference set is the only written record of what a team has agreed "correct" means, and in its absence every quality argument reduces to whose anecdote is more recent. It is also the asset with the longest useful life in the whole evaluation stack: harnesses get rewritten, metrics get replaced, judges get swapped for better ones, and the hundred hard cases a domain expert spent a week adjudicating stay valuable for years. The characteristic organisational failure is treating it as a by-product of tooling rather than as the thing the tooling exists to serve.

Why the expected answer is the hard part

Collecting inputs is easy; production supplies them in unlimited quantity. Stating what should come back is where the work is, and the form of that statement determines everything downstream. A single verbatim string is cheap to score and wrong for anything open-ended, because it rejects every acceptable paraphrase. A rubric is honest about there being many right answers and needs a judgement to apply. A set of required and forbidden assertions sits between the two and takes real effort per item. Most sets fail not because the inputs were unrepresentative but because somebody wrote down one acceptable output for a question that had forty.

Key decisions

• Who is authoritative about the expected behaviour for each item, and whether that is the engineer, a domain expert, a support lead, or a policy document. An item whose authority is unrecorded cannot be revisited when someone disputes it.
• What shape the expectation takes per item — exact value, required assertions, forbidden content, a rubric, or a reference answer used only for comparison — since the set will need several and a uniform schema will fit none of them well.
• How composition is argued. Proportional to traffic makes the aggregate meaningful and buries the rare high-severity cases; weighted toward difficulty makes every number pessimistic and more useful for finding defects.
• Whether an item's expectation can change, and what happens to the historical scores that were computed against the old one.
• How items are sourced: written from the specification, mined from production, distilled from support tickets, or generated. Each produces a different kind of blindness, and a set from one source alone inherits it completely.
• What each item carries besides input and expectation — intent class, severity, language, source, date added — because without those fields the set can only ever produce one number.
• Whether material drawn from production can live in the set at all, given consent, retention and the possibility that a customer's document is now a permanent fixture of your test suite.

Failure modes

• Items are collected from logged requests that the system answered well, so the set is a record of what already works and awards a perfect score to a model that has learned nothing.
• An expectation is a single exact string, the model returns a correct answer phrased differently, and the team spends a month chasing a regression that is a formatting difference.
• Half the set came from one enthusiastic contributor's domain, and the aggregate score is effectively a measurement of that one area.
• The hardest and most valuable items are removed because they fail constantly and make the dashboard look bad.
• Nothing records when an item was added, so a suite that has not grown in eighteen months looks identical to one curated last week.
• Expected answers encode facts that have since changed — a price, a policy, an interface — and the set now penalises the system for being right.
• A customer document included as an input is deleted at that customer's request everywhere except the evaluation repository.
• Three near-duplicate phrasings of the same underlying question are counted as three items, and a single failure mode registers as three times its real weight in the aggregate.

Related capabilities

• Evaluation Dataset Versioning — how a particular state of this collection is named and retrieved
• Offline Evaluation — the harness that consumes these items
• Human Evaluation — where contested expected answers get adjudicated
• Adversarial Testing — a deliberate source of items that curation from traffic will never produce
• Regression Evaluation — the consumer that depends on the set's composition staying stable

Applies: Production readiness · Design questions