Evaluation Gates — Patterns

Derive the level from variance and a stated consequence

Problem. Thresholds get set to a round number near the current score. They then fire on ordinary fluctuation, get edited under deadline pressure, and within two quarters the level tracks whatever the system last scored — which means the gate can only ever detect a change it has already accepted.

Approach. Set each level from two inputs recorded with it: the measured run-to-run variance of that metric on that suite, which establishes the smallest movement worth reacting to, and a written statement of the consequence the level is protecting against. Record who approved it and a date for review. Changing a level requires the same review as changing the system, with the new evidence attached.

Trade-off. It makes thresholds slow to set and slow to change, which is the point and is genuinely obstructive when a level turns out to be wrong. Variance-derived levels are also often looser than people expect, and publishing that the suite cannot detect a small regression is an unwelcome but honest consequence.

Grade the consequence to the condition

Problem. When every condition blocks, the only available response to any failure is an override, and overriding becomes routine. A missing optional field and a severe safety finding get the same treatment, and the team learns that the control is noise.

Approach. Assign each condition a consequence proportionate to what it protects: a hard block with no override for the small number of absolute conditions; a block requiring a named second approver for serious quality regressions; a tracked exception with an owner and a due date for moderate ones; a capped rollout percentage where the measurement is better taken on live traffic anyway. Report which tier fired, so the release conversation starts from the severity rather than from the count.

Trade-off. Tiering is a judgement that invites lobbying, and conditions drift downward in tier over time as teams find them inconvenient. The tiers also need a review mechanism of their own, which is a control on a control and the first thing to be skipped.

Fail on a missing or invalid result

Problem. The suite errors, the dataset version cannot be resolved, the judge endpoint is unavailable. The gate receives nothing, interprets nothing as no failure, and lets the release through — and because nothing alerts, this can continue for weeks while the control is believed to be working.

Approach. Make the gate require a positive, well-formed result: a run that completed, against a resolvable dataset version, with the expected scorers and an item count within a stated tolerance of the previous run. Any deviation is a failure of the gate itself, reported distinctly from a quality failure. Emit a signal when a condition has not been evaluated within its expected interval, so a silently disabled gate becomes visible rather than merely absent.

Trade-off. Infrastructure flakiness now blocks releases, which is correct and will be resented, and the pressure to add a bypass for "evaluation errors" reintroduces exactly the hole the pattern closes. The item-count tolerance also conflicts with legitimately growing a dataset, which then requires a deliberate acknowledgement.

Make every override expire and leave a record

Problem. An override is used once for a genuine emergency. Nothing requires it to be revisited, the regression it permitted is never fixed, and the next release inherits a passing gate because the exception is still in force. The control is now disabled in a way that looks like compliance.

Approach. Require every override to carry an identified approver, a written reason, and an expiry date after which the condition applies again automatically. Create a tracked item at the moment of the override, owned by a person rather than a team, referencing the specific result that failed. Publish the count of active overrides as a standing metric, since a rising count is the clearest early indicator that the gating programme is failing.

Trade-off. Expiry turns a one-off decision into recurring work and will sometimes block a later, unrelated release at a bad moment. The active-override count is also easily gamed by re-approving on expiry, which looks like diligence and changes nothing.