Evaluation and Quality Engineering
Safety Evaluation — Stack
Safety evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The components supply cases, classify outputs against a taxonomy, and produce a reportable profile. The decisive questions are whose taxonomy it is and whether the classifier has been checked against people who know the policy.
- Taxonomy alignment. Whether the categories can be mapped onto your written policy, since a classifier reporting against its own categories measures somebody else's standard.
- Severity representation. Whether outputs are graded rather than labelled present or absent, because the reporting needs worst case as well as rate.
- Over-refusal coverage. Whether benign cases that resemble prohibited ones are available or constructible, since almost no public set includes them in proportion.
- Language and locale coverage. Whether cases and classifiers work beyond English, and whether classifier quality is reported per language rather than overall.
- Adjudication support. Whether contested cases can be routed to people, recorded with a rationale, and re-used as precedent.
- Validation evidence. Whether the classifier's agreement with human adjudication on your own cases can be measured, and whether the tool makes that easy.
- Handling of harmful content. Whether sensitive material can be stored, displayed and retained under the controls your obligations require.
- Standard mapping. Whether results can be expressed against an external framework an auditor or customer will recognise.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Published safety benchmark suites and harm datasets | Broad category coverage immediately, and comparability with published results | Someone else's taxonomy and someone else's severity; widely used sets are also widely trained on |
| Harm classifiers published by model vendors and safety projects (Llama Guard-family models, hosted moderation endpoints and comparable) | Scoring large volumes against a defined category set at low cost | Category definitions are fixed and rarely match your policy; per-language quality varies sharply and is seldom reported |
| Judge models applied to a hand-written policy rubric | Measuring against your actual standard, including categories no public classifier covers | Needs validation against human adjudication before any rate is reportable; a judge's notion of a category is not your policy |
| Over-refusal and benign-prompt test sets | The counterweight measurement, which almost nothing else supplies | Small and rapidly saturated; their notion of benign may not match your product's risk appetite |
| Red-team finding repositories feeding this measurement | Populating the contested middle, which policy-derived cases rarely reach | Findings are shaped by whoever attacked; coverage is unquantifiable and biased toward recent interest |
| Annotation platforms with reviewer-welfare controls (blurring, exposure limits, rotation) | Human adjudication of harmful material, done responsibly | Welfare controls are features that must be configured, not defaults; throughput targets undermine them |
| Governance and assurance platforms mapping results to external frameworks | Organisations that must evidence compliance against a named standard on a schedule | Mapping produces documents rather than measurement; the underlying evaluation quality is unaffected by the mapping |