Golden Datasets — Patterns

Express the expectation as assertions, not as an answer

Problem. For anything open-ended there is no single correct output, so storing one forces a false choice: score by exact match and reject correct paraphrases, or score by similarity to the stored text and reward imitation of its style rather than its content.

Approach. Record what must be true of an acceptable response and what must not appear in one: the figures it has to state, the entity it has to name, the caveat it has to include, the competitor it must not mention, the advice it must not give. Each assertion is independently checkable, by string matching where the claim is literal and by a judgement where it is not. The item's verdict becomes a conjunction over assertions, which localises a failure to the specific requirement that was missed.

Trade-off. Assertions cost far more to author than a reference answer, and the set is only as good as the imagination of whoever enumerated them — an output can satisfy every assertion and still be unusable. The list also needs maintenance as requirements change, and a stale forbidden assertion is the most confusing kind of test failure.

Seed from incidents, not from averages

Problem. A set assembled by sampling traffic is dominated by the easy centre of the distribution. It produces a comfortable score, moves hardly at all between releases, and misses the cases that generate complaints — which are rare by definition and therefore almost absent from any proportional sample.

Approach. Treat every production incident, support escalation and internally reported bad answer as a candidate item, with the correct behaviour written at the time the incident is understood and the reasoning still available. Keep these in an explicitly labelled severity stratum, report it separately from the representative stratum, and never let an item leave it because it fails too often. The two strata answer different questions and should never be averaged into one figure.

Trade-off. An incident-seeded stratum is a record of past failures and says nothing about new ones, so a system can pass all of it and still be getting worse. It also grows monotonically unless somebody prunes, and pruning is politically hard because every item is attached to a remembered outage.

Give every item a stable identity and an audit trail

Problem. Items get edited, reworded, reordered and deduplicated. Without identity, a score history is a sequence of numbers computed over quietly different populations, and "this item started failing in March" is unanswerable.

Approach. Assign each item an identifier at creation that never changes and is independent of its position or content. Record alongside it who added it, when, from what source, who last changed the expectation and why. Treat a material change to an expectation as creating a new item that supersedes the old one, rather than as an edit, so historical results stay attached to the expectation they were judged against.

Trade-off. Supersession grows the collection with items that are no longer active and complicates every query with a status filter. The audit fields also need enforcing, and a set where half the items have an unknown author is worse than one with no audit fields, because it implies a rigour that is not there.

Keep a small adjudicated core separate from the bulk

Problem. Large sets drift in quality. Items get added in bulk, generated, imported, and never reviewed, until nobody is confident that a failure indicates a defect rather than a wrong expectation — at which point the whole set stops being usable as evidence.

Approach. Maintain a deliberately small core whose every item has been reviewed by a named domain authority and whose expectations are trusted without question, and a larger surrounding body that is useful for breadth but is understood to contain errors. A failure in the core is a defect and blocks; a failure in the bulk opens an investigation that may conclude the item was wrong. Items are promoted into the core only through review, never by accumulation.

Trade-off. Two tiers mean two sets of reporting and a standing argument about which tier a given item belongs to. It also concentrates authority in whoever reviews the core, and when that person leaves, the tier's guarantee quietly expires while its reputation does not.