Model Routing

What it is

The decision, made before a call is issued, about which of several available models should answer this particular request. The decision is driven by properties of the request — declared task type, input size, tenant tier, required modality, latency target, cost ceiling — not by anything having gone wrong. Routing is a policy; executing it is somebody else's job.

Why it matters

Traffic is not uniform. In most products a minority of requests genuinely need the most capable model available, and the majority are classification, extraction, reformatting, or short answers that a far cheaper model handles indistinguishably. Sending everything to the strongest model overpays on the bulk of traffic and adds latency to requests that did not need the thinking; sending everything to the cheapest model fails on the tail that did. Routing is how a product sits somewhere other than those two corners.

Why it is not fallback

A route is chosen on the way in, from the properties of a healthy request. A fallback is chosen on the way out, from the properties of a failure. A system can route without ever falling back, and can fall back from a route it chose deliberately.

Key decisions

• What the decision is a function of. Declared intent from the caller is cheap and honest but requires callers to be right; an inline classifier is accurate and adds a model call to the latency of every request it routes.
• Whether a route is sticky for the lifetime of a conversation or re-evaluated per turn. Re-evaluating mid-thread changes voice, formatting habits, and tool-call style.
• Where the cost-quality threshold sits, and against what measurement. Without a task-level evaluation set, "good enough for this request class" is an assertion rather than a finding.
• Whether the routing decision and its inputs are recorded on the request, so a later quality regression can be attributed to a route rather than to the product.
• How a newly admitted model earns traffic: manual table edit, gradual percentage ramp, or automatic promotion from measured results.

Failure modes

• The classifier that picks the model costs more latency than the difference between the models it is choosing between, and nobody measures the router's own overhead.
• Input length is used as a proxy for difficulty. Short, hard questions route to the small model and fail; long, trivial pastes route to the expensive one and waste money.
• Routing is correlated with request difficulty, so any A/B comparison of the two models on routed traffic is confounded and will favour whichever model got the easy half.
• The cheap path receives no evaluation coverage because the eval suite was built against the default model, and its quality decays unobserved for months.
• A mid-conversation route change drops the implicit context the previous model had established, and the user sees an abrupt change in tone or a repeated question.
• Routing rules accumulate as special cases for individual tenants until no one can predict which model will answer, and the table cannot be changed safely.

Related capabilities

• Multi-model Support — without it there is nothing to route between
• Model Fallback — the separate decision taken after a call fails
• Model Capability Registry — supplies the facts a routing rule tests
• Model Benchmarking — produces the evidence a routing threshold needs
• Semantic Caching — a cache hit removes the routing decision entirely

Applies: Production readiness · Design questions