Guardrails and AI Safety

Guardrails and AI Safety

Fail-closed Behavior — References

Fail closed behavior › References

56 of 57 in this book · 2 min read

The security principle that names the default, the reliability literature on degradation under load, and the accident research on why protective layers go missing without anyone noticing.

The default

  • The Protection of Information in Computer Systems — Jerome H. Saltzer and Michael D. Schroeder, Proceedings of the IEEE, 1975. Cited here for fail-safe defaults: access decisions should rest on permission rather than exclusion, so that a failure of the deciding mechanism produces refusal rather than access. The original statement of the choice this capability is about. Cited by name.

Degradation under load

  • Site Reliability Engineering — the published Google material, relevant here for graceful degradation, load shedding, and the treatment of a dependency's availability as a hard ceiling on the availability of anything that requires it, which is the arithmetic behind making a check mandatory.
  • The Tail at Scale — Jeffrey Dean and Luiz André Barroso, Communications of the ACM, 2013. Why latency outliers dominate at scale and why timeouts and hedged requests behave differently from their averages, which is the reason a check's p99 rather than its mean determines how often its posture is actually exercised. Cited by name.
  • Metastable Failures in Distributed Systems — Nathan Bronson, Abutalib Aghayev, Aleksey Charapko and Timothy Zhu, Workshop on Hot Topics in Operating Systems, 2021. Systems that remain in a degraded state after the triggering condition has passed, which describes what happens when a check's failure mode generates retries that sustain the overload that caused it. Cited by name.
  • Fail at Scale — Ben Maurer, ACM Queue, 2015. An operational account of how large systems fail, including the observation that overload responses and retry behaviour are themselves common causes of outages rather than remedies for them. Cited by name.

Why a missing layer goes unnoticed

  • How Complex Systems Fail — Richard I. Cook, 1998. The observation that complex systems run in a degraded mode continuously, with latent failures present and unremarked, so the absence of incidents is evidence about reporting rather than about defences. The most compact statement of why a zero violation rate needs to be investigated rather than celebrated. Cited by name.

No verified reference is recorded for three-valued verdict propagation in safety checks, for declared postures at check registration, or for measured availability of guardrail components in deployed language-model systems. The reliability literature treats a failed dependency as a correctness or latency problem and does not consider the case where the dependency is the thing enforcing a rule.