Model Management and Inference
Model Fallback — Stack
Model fallback › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
Resilience machinery is mature; what is immature is applying it to a dependency that can fail while returning 200 OK.
- Error classification depth. Can the mechanism distinguish a
429with a retry hint from a500, a hard timeout from a stalled stream, and an HTTP success carrying an unusable body? A library that only sees status codes sees about half of these. - Budget awareness. Whether the retry policy is bounded by a deadline propagated from the caller, not only by an attempt count.
- Jitter. Whether backoff is randomised by default. Deterministic backoff across many clients produces synchronised retry waves.
- Circuit-breaker granularity. Per model, per provider, per region — a breaker that trips for a whole provider when one model is degraded removes healthy capacity.
- Streaming semantics. What happens to an emitted prefix when an attempt fails mid-stream, and whether the mechanism can express "do not retry, we have already spoken".
- Idempotency support. Whether an idempotency key or attempt id is carried to the upstream, so a duplicate attempt can be recognised rather than re-executed.
- Observability. Per-attempt records, not just a final outcome; without them the fallback rate is invisible until the bill or the quality review.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Provider SDK built-in retries | Getting correct 429 and Retry-After handling immediately |
Defaults are per-call and unaware of your deadline; they compound with any outer retry you add |
Resilience libraries (resilience4j, Polly, Tenacity, backoff) |
Expressing retry, breaker, bulkhead, and timeout policies explicitly in application code | They classify on exceptions; a 200 response with a refusal or broken JSON needs a predicate you write |
| Proxy-level retries and outlier detection (Envoy, service mesh) | Uniform transport-level resilience across many services with no code change | Blind to response content and token cost; retrying a long generation at the proxy doubles the spend invisibly |
| LLM gateway fallback chains (LiteLLM Proxy, Portkey and similar) | Declaring an ordered model chain once, centrally, with per-model breakers | The chain's policy is now infrastructure configuration; make sure product teams can see which link answered |
| Durable workflow engines (Temporal, AWS Step Functions, Restate) | Long-running agent steps where an attempt must survive a process restart | Heavier operationally; the retry is durable, which makes an un-idempotent tool call worse, not better |
| Queue-based deferral with dead-letter handling | Batch and asynchronous workloads where a slower answer is acceptable | Useless for interactive turns; the user is still waiting |
A degraded-but-honest response path is part of this stack even though no product sells it: decide in advance what the system says when every link in the chain is exhausted.