Model Fallback — Patterns

Classify before reacting

Problem. A single except branch or a single retry decorator turns every unsuccessful outcome into the same behaviour. Rate limits, stalled streams, refusals, and malformed JSON have nothing in common except that the caller did not get what it wanted.

Approach. Normalise every outcome into an explicit category before any reaction is chosen — transient transport, throttled with a hint, upstream degraded, deadline exceeded, unusable output, permanently invalid request, policy refusal. Attach the category to the attempt record. The reaction table is then a small mapping that can be read, tested, and argued about, rather than control flow spread through exception handlers.

Trade-off. The categories are provider-specific at the edges and need maintenance as providers change their error bodies. Misclassification is now a single, findable bug rather than diffuse behaviour — which is the point, but it does concentrate the risk.

Deadline propagation across the chain

Problem. Per-attempt timeouts are set locally and multiply. Three attempts at ten seconds inside a caller that gives up at fifteen means the third attempt is never made and the second is abandoned mid-flight, having already been paid for.

Approach. Pass a single remaining-time budget into the attempt chain. Before each attempt, check that the remaining budget exceeds a realistic completion time for that target; if it does not, skip straight to the degraded response rather than starting a call that cannot finish. Shrink the budget by elapsed time, not by a fixed decrement.

Trade-off. Requires every layer to honour and forward the deadline, including library code that prefers its own defaults. It also makes the system refuse to try things it has time to try if the per-target completion estimates are pessimistic.

Commit-point-aware stream recovery

Problem. An attempt that fails after streaming 400 tokens to the user cannot simply be replaced by a fresh attempt: the user has already read the prefix, and a second attempt produces a different one.

Approach. Define an explicit commit point. Before the first byte reaches the user, any attempt may be abandoned and replaced freely. After it, the only permitted recoveries are in-place continuation or an honest in-band message that the response was interrupted. Buffer a short opening window — enough to catch immediate failures — and accept that failures after it are visible.

Trade-off. The buffer trades time-to-first-token for recoverability, and the window is a tuning parameter with no good universal value. Products that stream aggressively will occasionally show users a truncated answer and must have wording ready for it.

Idempotent tool execution across attempts

Problem. A failed agent turn may have already called a tool that sent an email, charged a card, or created a record. Re-running the turn re-runs the side effect.

Approach. Make the unit that gets retried the model call, not the turn: execute tool calls outside the retried region, keyed by a deterministic idempotency key derived from the turn id and the call's arguments. On a retry, a repeated key returns the recorded result instead of re-executing. Tools that cannot support a key are marked non-retryable and their turns fail closed.

Trade-off. Requires a durable store for the executed-call records and discipline from every tool author. Marking a tool non-retryable reduces availability honestly, which is usually better than silently duplicating an effect, but it is still reduced availability.