Safety Evaluation

What it is

Measuring how often, and how badly, a system violates a written standard of acceptable behaviour — and, equally, how often it refuses things the standard permits. It requires a policy specific enough to be tested, a taxonomy of the categories that policy covers, a severity scale that distinguishes an awkward answer from a dangerous one, and a population of cases spanning clear violations, clear permissions and the contested ground between them. The result is a profile of where the system sits against the standard, not a verdict that it is safe.

Where measurement ends and enforcement begins

Runtime guardrails make a decision about one response: block it, rewrite it, escalate it, or let it through. They are part of the system, they operate under a latency budget, and their job is to prevent a specific bad output reaching a specific user. This capability does none of that. It takes a population of cases and produces a rate, measured before release and over samples afterwards, and its unit of output is a distribution rather than a decision. The two depend on each other in one direction: a guardrail's threshold is only defensible if somebody measured what it does to the two populations it separates, and that measurement is this. Conversely, a safety profile measured with the guardrails disabled describes a system nobody ships, so the configuration under test has to be stated explicitly and is itself a decision.

Why it matters

Harm is the failure class where a rate of one in ten thousand is still unacceptable, which makes it the hardest thing in the stack to measure — rare events need large samples, and the cases that matter most are the ones nobody has thought of. It is also the only area where the counterweight is as important as the metric: a system that refuses everything scores perfectly on violations and is useless, and a system tuned only against violation rate will drift there. Where regulation or contractual commitments apply, the measurement is also the evidence, and an evidence trail assembled after an incident is worth considerably less than one produced on a schedule.

Key decisions

• What the standard is and where it is written, since the measurement is only as specific as the policy — "do not produce harmful content" cannot be tested, and a per-category rule with examples can.
• What the category taxonomy is, and whether it is adopted from an external framework or written for the product, given that the latter will not map to anything regulators or auditors expect.
• How severity is assigned, and whether reporting is by rate or by worst case. A rate treats a thousand mild violations as worse than one severe one, which is almost never right.
• Whether over-refusal is measured with the same rigour as violation, including the benign cases that superficially resemble prohibited ones.
• What configuration is evaluated: the model alone, the model with its instructions, or the full system with runtime enforcement in place. Each answers a different question and only the last describes production.
• How the case population is sourced, and in what proportion between policy-derived cases, production incidents and findings from deliberate attack.
• Who adjudicates a contested case, and how that adjudication is recorded so the same argument is not relitigated every quarter.
• How reviewers exposed to harmful material are protected, which is an obligation rather than a throughput consideration.

Failure modes

• The policy is a paragraph of principles, every contested case is argued from first principles, and two evaluations months apart measured different standards.
• Violation rate falls and nobody measures refusals, so the improvement is the system declining an increasing share of legitimate requests.
• A single rate is reported across categories whose consequences differ by orders of magnitude, and a movement in the mildest category masks a rise in the most severe.
• The evaluation set contains only blatant cases, the system passes, and everything in the ambiguous middle — where nearly all real traffic sits — is unmeasured.
• Measurement is taken against the model with no instructions or enforcement, and the resulting profile is published as though it described the product.
• Cases are English-only, and the behaviour in the other six languages the product serves is entirely unknown.
• The set is frozen at launch, the policy is revised twice, and the suite keeps reporting against the superseded standard.
• Judge-based scoring of harm is never validated against human adjudication, and the reported rates rest on a model's unexamined notion of the category.
• Reviewers work through harmful material without rotation, exposure limits or support, and the programme is sustained by people it is damaging.

Related capabilities

• Adversarial Testing — the deliberate search whose findings populate the hard end of this measurement
• Human Evaluation — adjudicates contested cases and validates every automated harm scorer
• Golden Datasets — holds the case population, its categories and its severity labels
• Evaluation Gates — where a severity ceiling becomes a release condition
• Content Moderation — the runtime decision on a single response, whose operating point this measurement informs

Applies: Production readiness · Design questions