Safety Evaluation — Patterns

Report violation against refusal, always as a pair

Problem. Optimising a violation rate alone has one guaranteed solution: refuse more. Systems tuned against that single number become progressively less willing to discuss medicine, law, security, history and anything else adjacent to a prohibited category, and the degradation is invisible to the metric driving it.

Approach. Maintain two populations of equal care: cases that must be declined or handled cautiously, and benign cases that superficially resemble them — the security question asked by a defender, the clinical question asked by a clinician, the historical account of an atrocity. Report the violation rate on the first and the refusal rate on the second as a single joint figure, and require that a change improving one does not worsen the other beyond a stated bound.

Trade-off. The benign population is much harder to construct than the prohibited one and saturates quickly, so the counterweight weakens over time while the violation set stays adversarially fresh. Deciding what counts as benign is also a product judgement that different stakeholders will answer differently, and the set encodes whichever answer was current when it was written.

Report the worst case alongside the rate

Problem. A mean over a taxonomy collapses categories whose consequences differ enormously. A release that reduces mild policy slips while introducing one case of serious harmful instruction will show as an improvement, and the number that moved is not the number that matters.

Approach. Grade every observed violation on a severity scale defined by consequence rather than by category, and report per severity band: the count at the top band, the rate at the middle, the trend at the bottom. Make the top band a count and never a rate, since dividing a catastrophic case by the suite size is a way of making it disappear. Any case in the highest band is a finding in its own right with a named owner, regardless of what the aggregate did.

Trade-off. Severity assignment is a judgement that drifts between assessors and over time, and a scale with a strict top band creates pressure to classify downward when a release is due. Counting rather than rating the top band also makes suite size changes confusing, since adding cases can only increase the count.

Derive cases from the policy clause by clause

Problem. Case sets assembled from available public data measure the categories those sets happen to cover. The product's own policy — the document people are actually accountable to — goes untested wherever it is more specific than a public taxonomy, which is usually everywhere that matters.

Approach. Walk the policy clause by clause and generate cases for each: a clear violation, a clear permission, and at least two in the contested region, with the expected handling written down and attributed to whoever is authoritative for that clause. Keep the clause reference on each case, so that a policy revision produces a list of exactly which cases must be re-authored and which results became invalid.

Trade-off. It is slow, requires sustained access to whoever owns the policy, and produces a set that only covers what the policy anticipated — the harms nobody wrote a clause about remain unmeasured. Clause-linked cases also make policy revision visibly expensive, which can discourage necessary revision.

Evaluate the shipped configuration, and say which one it was

Problem. Safety numbers get produced against whatever was convenient: a bare model endpoint, a model with the system instructions, or the full product with runtime enforcement. The three differ by a large factor, and the number travels into decks and customer conversations without the configuration attached.

Approach. Define the configuration as part of the result — model build, instruction revision, enforcement components and their thresholds, retrieval state — and report at least two: the deployed configuration, which is what the product actually does, and the model with instructions and no enforcement, which shows how much of the safety profile is carried by the guardrails. The gap between them is the measure of how exposed the product is if enforcement fails open.

Trade-off. Two configurations mean two runs on a suite that is already expensive to score and adjudicate, and the layered view invites the wrong conclusion that a large gap is fine because enforcement exists. Enforcement thresholds also change far more often than the suite runs, so the deployed figure is stale almost immediately.