Multi-model Support · Model Management and Inference

What it is

The ability to serve application traffic from more than one model, and to change which model answers a given request without changing calling code. It is distinct from Model Routing, which decides which model answers; multi-model support is the substrate that makes more than one answer possible at all.

Why it matters

A single model is a single point of failure with a price attached. Providers deprecate versions on their own schedule, rate-limit under load, and change refusal behaviour between releases. Teams that hard-code one model discover the coupling during an outage.

Key decisions

Whether the application speaks a normalised internal request shape or each provider's native API. Normalising costs a translation layer and hides useful provider features.
Where tokenizer differences are reconciled, since token counts are not comparable across families and budget logic silently drifts if it assumes they are.
Whether tool-call schemas are translated or restricted to a common subset.
How streaming differences are presented, given that chunk shapes and stop semantics differ.
Whether responses are recorded against the model and version that produced them.

Failure modes

A prompt tuned on one model regresses on another; nothing catches it because evaluation runs against only the default.
Token accounting is computed with one tokenizer and billed against another, so budget ceilings are wrong by a double-digit percentage.
A provider returns a refusal where another returned content, and the application treats the refusal as a transport error and retries it until the rate limit trips.
Structured-output parsing assumes one provider's JSON quirks and breaks on another's.

Related capabilities

Model Routing — chooses between the models this capability makes available
Model Fallback — what happens when the chosen model fails
Model Gateway — the usual place this abstraction is implemented
Model Capability Registry — records what each model can do

Applies: Production readiness · Design questions