Model Management and Inference
Model Routing — Stack
Model routing › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
What you are buying is a decision function and the evidence that it is decided well.
- Decision latency. Added milliseconds at p95 before the chosen model is even called. A router that costs 300 ms to save 200 ms is a loss.
- Explainability. Can the system tell you, for a specific past request, which rule or score chose the model and what the inputs were? Without this, quality incidents are unattributable.
- Calibration data. Does the mechanism need labelled traffic to work, and do you have it? Learned routers are only as good as the task labels behind them.
- Change safety. Whether a routing change can be ramped by percentage, scoped to a tenant, and rolled back without a deploy.
- Model-set churn. How much work it is to add a model to the table — a configuration line, or retraining a classifier.
- Interaction with stickiness. Whether the mechanism can honour a per-conversation pin, which most learned routers do not do natively.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Static rules over declared task type | Products where the calling code already knows what it is asking for | Depends entirely on callers declaring honestly; drifts as new call sites appear |
| Heuristic rules on request features (length, modality, tenant tier) | Fast first implementation with zero extra model calls | Length is a weak proxy for difficulty; rules accrete tenant-specific exceptions |
| Small classifier model as a router (fine-tuned encoder or small LLM) | High-volume traffic with a stable, labelled task taxonomy | Adds a call to the hot path; needs retraining when traffic or the model set shifts |
Embedding similarity to labelled exemplar sets (semantic routers such as Aurelio's semantic-router) |
Intent-style routing where exemplars are easier to produce than labels | Thresholds need periodic recalibration; unseen intents land on whichever class is nearest |
| Cascade: cheap model first, escalate on a confidence or verification signal | Tasks with a cheap, reliable check on the cheap model's answer | Two calls on the escalating fraction; the whole design rests on the quality of the check |
| Gateway-level routing policy (LLM gateways and API gateways with model-selection plugins) | Enforcing one routing policy across many services without redeploying them | Routing decisions made in infrastructure are harder for product teams to inspect and test |
| Managed router products from model vendors and platforms | Getting a cost-quality curve quickly without owning the machinery | The decision boundary is opaque, vendor-scoped, and moves without your release process |