Model Routing — Patterns

Declared task class as the routing key

Problem. Inferring what a request needs from its raw text requires a model, and that model is on the critical path of every request. Yet the calling code usually already knows whether it is summarising, extracting fields, or answering an open question.

Approach. Make task class an explicit, enumerated field on the internal request, set at the call site. Routing becomes a lookup from task class and tenant tier to a model, held in configuration. Unknown or missing classes route to a defined default and raise a metric rather than being silently guessed.

Trade-off. The taxonomy must be maintained, and a wrong declaration is invisible — the request is routed confidently to the wrong place. It also pushes a product concern into every call site, which is exactly the coupling an abstraction layer tried to remove.

Cascade with a verification gate

Problem. For many tasks the cheap model is right most of the time, but which times is not predictable from the input alone.

Approach. Call the cheap model first, then apply a check that is strictly cheaper than the strong model: schema validation, a required citation resolving to a retrieved document, a numeric answer recomputed deterministically, or a self-reported confidence calibrated against held-out data. Escalate only on a failed check.

Trade-off. Escalated requests pay both calls plus the check, so the economics depend on the escalation rate staying low — and on the check being independent of the cheap model's failure mode. A check the cheap model can satisfy while still being wrong buys nothing.

Conversation-sticky assignment

Problem. Per-turn routing in a multi-turn product changes the answering model mid- thread. Formatting conventions, verbosity, and tool-call phrasing change with it, and the user experiences the product as inconsistent.

Approach. Resolve the model once at session start, record it on the session, and reuse it for subsequent turns. Allow a deliberate upgrade — a turn that needs a longer context or a modality the pinned model lacks — but treat it as an explicit, logged transition rather than a recomputation.

Trade-off. Long sessions stay pinned to a model that may since have been deprecated, repriced, or superseded, so sessions need a maximum pin lifetime and a defined migration when the pinned model is withdrawn.

Shadow-scored routing changes

Problem. A proposed change to the routing table looks better on the evaluation set, but the evaluation set is not the traffic. Shipping it is the only way to find out, and it is also how a regression reaches users.

Approach. Evaluate the candidate policy offline against recorded production requests, replaying the decision only — which model would have been chosen — and compare the resulting cost and model mix before any change to live traffic. Then ramp by percentage with the route recorded on every request, so the comparison can be made on real outcomes.

Trade-off. Replaying decisions is cheap, but it tells you about distribution, not quality: it cannot show that the cheap model would have answered worse. It narrows the risk; it does not remove the need for a ramp.