Evaluation Gates

What it is

A rule that converts an evaluation result into a release decision: a stated condition, a defined consequence when it is not met, and a named person who can override it. The substance is in the condition — which measurements are allowed to block, at what level, measured over what population — and in the override, because a rule nobody can set aside gets removed within a quarter and a rule anybody can set aside was never a rule.

What a gate owns and what it does not

Detecting that a number moved, and deciding whether the movement is real, is an analysis question answered before a gate ever runs; the gate consumes a result that has already been judged trustworthy. Wiring the check into a pipeline — when it executes, how it reports, how a build is held — is delivery plumbing that varies by toolchain and belongs with the pipeline. What is left, and what this capability is about, is the policy: which conditions are worth stopping a release for, how the level in each condition was arrived at and by whom, what happens on a failure, and how the rule is changed. A gate with a threshold nobody can justify is a number somebody typed, and it will be edited the first time it is inconvenient.

Why it matters

Without an enforced condition, evaluation is advisory, and advisory measurements lose to deadlines reliably. The gate is also what makes a quality claim institutional rather than personal: it survives the engineer who cared about it leaving, and it converts "we think this is fine" into a statement about a measurement that somebody can inspect afterwards. The opposite failure is equally common and more corrosive — a gate that fires on noise trains a team to override by reflex, and once overriding is routine the organisation has the cost of the control and none of its protection.

Key decisions

• Which measurements are allowed to block at all. A small number of conditions tied to consequences people agree on is enforceable; a gate on every metric produces a permanent amber state and reflex overrides.
• Whether a condition is absolute or relative — an output must never fall below this level, or must not fall by more than this much from the previous release. The two fail in different directions and most programmes need both.
• How the level is set and recorded. A threshold derived from measured variance and a stated consequence can be defended; a round number cannot, and will be argued away.
• Who may override, what they must record, and whether an override expires — since the usual end state of an un-expiring override is a permanently disabled gate nobody remembers disabling.
• What happens on a failure: block, require a second signature, ship with a tracked exception, or reduce the rollout. Not every condition deserves a hard stop, and treating them all alike is what produces reflex overriding.
• How the gate behaves when the evaluation cannot run — dependency outage, missing dataset version, scorer error — because the implicit answer is usually to pass, which is the worst one.
• How often thresholds are revisited, and against what, given that a level set at launch describes a system that no longer exists.

Failure modes

• The threshold is just below the current score, so the first ordinary fluctuation blocks a release and the number is edited that afternoon.
• Overrides are permitted with no recorded reason, and within two quarters every release carries one.
• The condition is set on an aggregate that noise moves by more than the threshold, so the gate's verdicts are effectively random.
• Fourteen metrics each have a gate, something is always failing, and the release conversation is about which failures to ignore.
• The evaluation errors out, the gate interprets no result as no failure, and releases flow unchecked for weeks before anyone notices.
• A threshold is absolute and the evaluation dataset grew harder, so the gate blocks a system that got better.
• A gate blocks, the release is urgent, the override is used, and nothing creates a follow-up — so the regression ships permanently and the gate's only effect was a delay.
• A severity-weighted condition is replaced by a mean because the mean was easier to compute, and a single catastrophic case no longer blocks anything.
• The gate runs against whatever dataset version is current, which changed that morning, and the comparison it blocks on was never valid.

Related capabilities

• Regression Evaluation — establishes that a movement is real before a condition acts on it
• Offline Evaluation — produces the results these conditions are stated over
• Safety Evaluation — the area most likely to warrant an absolute, non-overridable condition
• Evaluation Dataset Versioning — a threshold only means something against a stated version
• Deterministic Evaluation — the scorer family whose lack of variance makes a gate defensible

Applies: Production readiness · Design questions