Multi-model Support — Patterns

Normalised request envelope

Problem. Call sites accumulate provider conditionals. A change of default model means editing every place that builds a request, and each site drifts in how it handles system prompts, tool definitions, and stop sequences.

Approach. Define one internal request object — messages, tool declarations, output contract, generation parameters, budget ceiling — and one internal response object that carries content, tool calls, stop reason, usage, and the resolved model identity. Adapters translate the envelope into each provider's native call and translate the result back. Nothing above the adapter layer knows a provider name.

Trade-off. The envelope is a least-common-denominator by construction. Every provider feature outside it either forces an envelope change that all adapters must honour, or leaks through a passthrough field that quietly reintroduces coupling. Budget for the envelope to be revised a few times a year.

Capability probing at startup

Problem. Static configuration claims a model supports parallel tool calls or a 200k context; the deployed account, region, or tier actually does not. The mismatch surfaces as a production error on a request that mattered.

Approach. On service start — or on a schedule — issue a small set of cheap probe calls per configured model: a tool-call request, a structured-output request, a streaming request that is read to completion. Record the observed answers next to the declared ones and refuse to mark a model available until its probes pass.

Trade-off. Probes cost money and add startup latency, and they can only test what you thought to probe. They do not detect quality differences at all — only mechanical ones.

Shadow traffic for a candidate model

Problem. A second model is configured but unproven. Switching real traffic to it is the only way to learn how it behaves, and also the way to learn it in front of users.

Approach. Send a sampled copy of live requests to the candidate alongside the production model, discard its responses, and record both outputs against the same request id. Compare offline on the dimensions that matter for the product — refusal rate, output length, tool-call correctness, latency, cost per completed task.

Trade-off. You pay twice for sampled traffic and you must guarantee the shadow path can never emit side effects: no tool execution, no writes, no user-visible output. A shadow run that executes tools is an incident, not an experiment.

Provider-agnostic output contracts

Problem. One backend honours a strict JSON schema, another returns JSON wrapped in prose, a third returns a near-miss with a trailing comma. Parsing code written against the first fails open against the others.

Approach. Treat every model's structured output as untrusted text. Extract, parse, and validate against the schema in the application, independently of whether the provider claims to guarantee it; use the provider's native constrained decoding when available as an optimisation that reduces retries, never as the reason validation can be skipped.

Trade-off. Validation and repair add a round trip on failures and some latency on every call. In exchange, admitting a new backend does not require re-auditing every parser.