Fail-closed Behavior — Patterns

Three outcomes, never two

Problem. A check with a boolean return type has nowhere to put "I did not decide". Every infrastructure failure must therefore be coerced into either a violation or a pass, and since coercing to violation breaks the product under any transient fault, every implementation eventually coerces to pass. The blind spot is created by the type signature before anyone writes a policy.

Approach. Make the verdict a three-valued type at every boundary: allowed, denied, indeterminate. Indeterminate is never collapsed by the component that produced it — the caller decides what it means in context, and the decision is explicit code. Record indeterminate as its own outcome in metrics and traces, so its rate is visible beside the other two and an outage is immediately distinguishable from a quiet period.

Trade-off. Every call site now has a third branch to handle, and the branches multiply when several checks run against one request. The honest type also makes a previously simple interface awkward, and teams under delivery pressure will write a helper that maps indeterminate to allowed, which recreates the original problem with extra steps unless the helper is explicitly prohibited.

Serve the last known good, with an expiry

Problem. Refusing everything when the policy service is unreachable is a self-inflicted outage; carrying on with no policy is unenforced. Both are consequences of treating rules as something fetched per request rather than as something held.

Approach. Distribute rules, models, and thresholds as versioned artifacts that each evaluator caches locally with a declared maximum age. While fresh, the cached copy serves normally and the control plane's availability is irrelevant to the request path. Past the maximum age, staleness becomes a failure in its own right and the configured posture for that check applies. Emit the age of the artifact in use as a metric, so drift is observable before it becomes an incident.

Trade-off. A withdrawn rule keeps being enforced, and a newly added one is not, for up to the maximum age — which is a correctness gap deliberately traded for availability, and is exactly the wrong trade for a rule withdrawn for legal reasons. Choosing the age is also a judgement nobody wants to make, and the usual outcome is a value long enough that it never triggers.

Declare the posture where the check is registered

Problem. Postures that live in handler code are set by whoever wrote the handler, differ between call sites for the same check, and are impossible to audit. Asking the question "what happens if this check is down" requires reading every caller, which means the answer is unknown.

Approach. Require every check to declare its posture as part of registration — what indeterminate means, the deadline, the maximum staleness, and what degraded operation looks like — and have the framework enforce it so no caller can choose differently. The registry becomes the single document answering the question for the whole system, and adding a check without a posture is a startup failure rather than an omission.

Trade-off. A single declared posture cannot express a judgement that genuinely varies by context: the same moderation check may reasonably fail open on an internal drafting surface and closed on a public one. Supporting per-context overrides reintroduces the scattering the pattern was removing, so the registry ends up encoding contexts too, which is more structure than small systems need.

Separate the switch from the deploy

Problem. When a check starts misfiring — a bad model update, a rule with an unintended match, a threshold off by an order of magnitude — the damage accrues per request while the fix goes through a build, a test suite, and a rollout. The usual emergency response is to disable the whole safety layer, because that is the only lever anyone can pull quickly.

Approach. Give every check an independent runtime control that can disable it, loosen it, or move it to monitor-only within seconds, operated through a path that does not involve the build system. Make the control itself audited and time-bounded, so a check disabled during an incident expires back on rather than staying off indefinitely. Rehearse it, because an untested switch is a plan rather than a capability.

Trade-off. A fast disable is a fast way to remove protection, and it will be used under pressure by whoever is on call with the least context. The control plane carrying it also becomes a dependency of the safety layer, which means the posture question now applies recursively to the mechanism that answers the posture question.