Evaluation and Quality Engineering
Golden Datasets — Stack
Golden datasets › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
This is a curation and authoring problem, so the tooling is judged by who can contribute and how well the result survives contact with a scorer.
- Non-engineer authoring. Whether a support lead or domain expert can add and correct an item without a pull request, since they are the people who know the right answer.
- Heterogeneous expectation schema. Whether one collection can hold exact-match items, rubric items and assertion-list items without forcing all of them into the lowest common shape.
- Per-item metadata. Whether arbitrary labels can be attached and filtered on, because reporting by intent class and severity is the difference between one number and a diagnosis.
- Promotion from traffic. How much work it is to take a recorded production request, strip what cannot be retained, and turn it into an item with an expectation attached.
- Review workflow. Whether proposed items and changed expectations go through an approval step with an identified approver recorded on the item.
- Deduplication support. Whether near-identical items can be found, since a set mined from traffic accumulates them and they distort every aggregate computed over it.
- Export stability. Whether the collection can be emitted as a plain, diffable file that a harness reads directly, rather than only through an API that becomes a runtime dependency.
- Access control and redaction. Whether sensitive items can be restricted and whether deletion of a source record can be propagated.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Plain files in the application repository (JSONL, YAML, CSV) | Engineering-owned sets where review, diffing and history come free from version control | Non-engineers cannot contribute; merge conflicts on a shared file; no structure beyond what a schema check enforces |
| Spreadsheets as the authoring surface with an export step | Getting domain experts contributing within a day | Silent type coercion, no validation, no identity per row, and a copy that diverges from the exported version |
| Dataset features of LLM evaluation platforms (LangSmith, Langfuse, Braintrust and similar) | Promoting a production trace into a test item with one action, and reviewing items in the same place the runs appear | The set becomes vendor-resident; export fidelity and item identity across export are worth testing before committing |
| Annotation platforms (Label Studio, Argilla, Prodigy and comparable) | Sets whose expectations need structured human adjudication, multiple reviewers, or span-level labelling | Built around labelling campaigns rather than long-lived curated collections; the ongoing-maintenance workflow is yours to build |
| Dataset libraries and hubs with versioned tabular formats (Hugging Face Datasets, Parquet in object storage) | Large sets, programmatic slicing, and sharing across several teams or repositories | Weak fit for frequent small hand edits; review of a change is a data diff rather than a readable one |
| A relational table behind an internal admin interface | Organisations where the set must obey existing access control, retention and deletion machinery | Someone has to build and maintain the interface, and it will be nobody's priority after the first quarter |
| Synthetic item generation tooling (model-driven case writers, scenario expanders) | Filling obvious coverage gaps quickly and producing variations around a known hard case | Generated items inherit the generator's blind spots, and an unreviewed generated expectation is an assertion nobody checked |