Model Management and Inference

Model Management and Inference

Model Fallback

14 of 49 in this book · 3 min read

What it is

The reaction taken after an attempted inference call has failed or returned something unusable: retry the same target, move to a different one, degrade the response, or give up and surface an error. Everything about it is triggered by an observed failure, which is what separates it from a routing decision made in advance on a healthy request.

Why it matters

Inference endpoints fail in ways ordinary service dependencies do not. They return 429 under someone else's load, stall mid-stream without closing the connection, emit a content refusal that is a valid HTTP 200, and produce syntactically broken structured output that only fails at the parser. Each of these needs a different response, and a system with one undifferentiated retry loop will answer all of them the same way — usually by retrying something that will never succeed until a rate limit or a bill says stop.

Key decisions

  • The failure taxonomy. At minimum: retry the same target with backoff (transient transport error, 429 with a retry hint), move to another target (sustained upstream failure, hard timeout), re-prompt the same target (unparseable structured output), and fail immediately without retrying (malformed request, authentication failure, policy refusal that will reproduce deterministically).
  • The total time and token budget for the whole attempt chain, derived from the caller's deadline rather than from the per-attempt timeout. A chain of three ten-second timeouts inside a client with a fifteen-second deadline never reaches its last link.
  • What a degraded answer is allowed to be: a weaker model's answer, a cached answer, a template, a partial result, or an honest refusal — and whether the user is told.
  • Whether the alternate target is required to support the same features. A fallback that cannot call tools is not a fallback for an agent turn; it is a different failure.
  • When to stop trying at all — the circuit-breaker threshold, and what the system does for the duration the breaker is open.

Failure modes

  • Retry amplification: the primary slows down, every caller retries, and the retries are the load that finishes it off. Backoff without jitter synchronises the retries into waves.
  • The fallback target is capacity-planned for its normal share of traffic and receives all of it, so the failure moves rather than being absorbed.
  • A stream fails after emitting half the answer; the fallback call starts from the beginning and the user sees the first half twice.
  • A fallback attempt re-executes tool calls that the first attempt already performed, duplicating a side effect that was never idempotent.
  • Policy refusals are classified as errors and retried, raising cost and producing nothing — the content will be refused on every attempt.
  • Fallback succeeds so reliably that the primary's outage never pages anyone; the product runs for weeks on the weaker model and quality regressions are attributed to prompts.
  • A timeout tuned for a short completion fires routinely on a long or reasoning-heavy response, so successful generations are abandoned and paid for.

Related capabilities

Applies: Production readiness · Design questions