Model Management and Inference
Multi-model Support — Stack
Multi model support › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
Judge an abstraction layer by how little it lies about the providers underneath it.
- Request-shape coverage. Does the common envelope carry system prompts, multi-turn message roles, images and other modalities, tool definitions, and response-format constraints — or only plain chat completion?
- Tokenizer fidelity. Does the layer expose the per-family tokenizer, or does it estimate with one encoding and let the error accumulate into billing and truncation?
- Streaming parity. Can a single consumer read deltas, tool-call fragments, usage totals, and stop reasons from every backend without branching on the provider name?
- Tool-schema translation. Whether a declared function schema survives the round trip, and what happens to constructs one backend accepts and another rejects.
- Escape hatch. Whether provider-native parameters can be passed through unmodified when the common denominator is not enough, without abandoning the abstraction.
- Dependency weight. An abstraction that pulls in every provider SDK transitively becomes the slowest part of a cold start and the widest part of the attack surface.
Options
| Option | Best for | Watch out for |
|---|---|---|
| OpenAI-compatible Chat Completions as a de facto wire format | Teams that want one HTTP contract and many backends, including self-hosted servers that already emulate it | Emulation is partial — tool calling, logprobs, and usage reporting are where compatibility usually stops |
| LiteLLM | Python services wanting many providers behind one call with per-model cost metadata | Provider feature coverage varies by adapter; pin the version, since provider-specific behaviour changes between releases |
| LangChain / LlamaIndex chat-model interfaces | Applications already built on those frameworks' abstractions | The model interface arrives with a much larger framework; upgrades move more than the model layer |
| Vercel AI SDK | TypeScript and edge-runtime products needing uniform streaming to a browser client | Strongest on the streaming/UI path; server-side batch and evaluation workloads are less of a focus |
| Provider SDKs plus a thin in-house adapter | Teams using two or three models that want exact control over each native API | You own the translation, the tests, and the day a provider changes its response shape |
| Cloud vendor model catalogues (managed multi-provider endpoints) | Organisations where procurement, residency, and billing must stay inside one cloud account | The catalogue's model list and feature support lag the model vendors' own APIs |
Models
Model choice is in scope here because the abstraction is only as good as the differences it has to hide. A practical starting set spans three behavioural classes rather than three brands: a high-capability reasoning model for the hard tail of traffic, a mid-tier general model as the default, and a small fast model for classification and extraction. Choose the set so that the capability gaps are known and tested, not so that the names look comparable in a pricing table. Verify for every model you admit: native tokenizer, maximum output length, whether tool calling and strict structured output are supported, and whether streaming reports token usage. Any model whose answers to those four questions are unknown is not yet supportable, however good its benchmark scores.