Guardrails and AI Safety

Guardrails and AI Safety

Human Approval Gates

50 of 57 in this book · 3 min read

What it is

The machinery that suspends a run at a defined point, puts a specific pending action in front of a specific person, and resumes or abandons the run according to their answer. It covers the durability of the paused state, what the approver is shown, how the request reaches them, what happens when nobody answers, and what record the decision leaves. Which actions require a person is decided elsewhere, by autonomy policy and by risk valuation; this is what makes acting on that decision affordable rather than catastrophic.

Why it matters

A pause is a control only if the thing resumed is the thing approved, and that is a distributed-systems problem before it is an interface problem. The run holds context, partially completed work, and sometimes an open transaction; the world it was acting on keeps moving; the person deciding may be asleep in another timezone. Products routinely get the dialog right and lose the guarantee. An approval that is cheap to request and expensive to honour degrades into one of two outcomes — a rubber stamp that documents consent while providing no oversight, or a queue deep enough that people route around the system entirely.

Key decisions

  • Where the paused state lives and what it contains. Holding it in the serving process means every deployment silently discards whatever was waiting.
  • Whether the approved payload is pinned exactly — arguments, model version, prompt version — so that what executes cannot differ from what was shown.
  • Which preconditions are revalidated at resume rather than at request: the balance, the record's state, and the approver's own entitlement at the moment of execution.
  • Timeout semantics. Expiry, escalation, or indefinite hold, and who is notified as a deadline approaches.
  • Routing: a named person, a rota, a group queue, or the requesting user themselves, and what happens outside working hours in every timezone the product serves.
  • The expected firing rate. This is a control only while each interruption is read, and past some rate per approver per day it is a tax that produces consent without attention.
  • Whether approvals may be batched or standing, and what bounds such a grant in scope, amount, and time.
  • What the record holds, so that months later it can be shown who approved precisely what, under which policy, with what shown to them.

Failure modes

  • The run is suspended, the underlying record changes while it waits, and the approved action executes against a state nobody assessed.
  • Pending items are held in memory, a routine deployment discards them, and the people waiting are never told that their request evaporated.
  • Resume re-executes steps that completed before the pause, because the suspension point was not idempotent and the replay had no way to know.
  • The queue has no owner and no service level, median wait exceeds the user's patience, and the agent stops being used for anything that triggers a gate.
  • Approvers are the engineers who built the system, so the review adds no independent judgement and does not survive the pilot.
  • An approval granted for one action is reused as consent for a retry whose arguments were regenerated and differ.
  • The request goes to a shared channel, is answered by whoever is awake, and the record attributes the decision to a group rather than to a person who can be asked about it.
  • A standing approval has its scope written in prose and no expiry, and is still authorising work a year after the project that justified it ended.
  • Escalation on timeout reaches someone with less context than the original approver, who approves because declining an unfamiliar request looks obstructive.
  • The approver is shown a rendering that omits a field the executor uses, so the part that mattered was never on the screen.

Related capabilities

Applies: Production readiness · Design questions