Model Management and Inference
Model Gateway — Stack
Model gateway › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
This is an infrastructure purchase, not a library choice. Evaluate it the way you would evaluate any proxy that sits in front of revenue-bearing traffic.
- Streaming passthrough. Does it forward server-sent events incrementally under load, and what is the added time-to-first-token at the p95 you actually serve?
- Credential isolation. Are provider keys held in a secret store and never exposed to callers, and can a single tenant's key be rotated without a restart?
- Per-caller accounting. Can it attribute tokens and spend to a tenant, team, and request id, and expose that as metrics rather than only as a billing export?
- Enforcement primitives. Rate limits, concurrency caps, and spend ceilings that apply per key and per tenant — and a defined behaviour when a ceiling is hit.
- Failure behaviour. What it does when an upstream is slow rather than down: queue, shed, or hang. Hanging is the common and worst answer.
- Operational surface. Configuration reload without dropping in-flight streams, readiness separate from liveness, and metrics in a format your platform already scrapes.
- Extension model. Whether custom policy is a plugin in a language your team writes, a sidecar, or a fork.
Options
| Option | Best for | Watch out for |
|---|---|---|
| General-purpose proxies (Envoy, NGINX, HAProxy) with custom filters | Platforms that already run one of these and need credential termination plus rate limiting | No model awareness: token counting, usage extraction, and per-model quotas are yours to build |
| API gateways (Kong, Apigee, AWS API Gateway, Azure API Management) with AI plugins | Organisations whose API governance, keys, and audit already live in that gateway | AI-specific plugins are newer than the platform around them; verify streaming behaviour before committing |
| Purpose-built LLM gateways (LiteLLM Proxy, Portkey, Helicone, Cloudflare AI Gateway) | Teams wanting model-aware routing, caching, and spend tracking without building them | Each is also a dependency on the critical path and a copy of your prompt traffic; check retention settings first |
| Service-mesh egress policy (Istio, Linkerd) | Estates where the requirement is residency and egress control rather than model features | Mesh policy operates on connections, not on tokens or prompts |
| In-process client library with shared configuration | Small estates, or latency budgets that cannot absorb an extra hop | Enforcement is advisory: anyone with the key can bypass it, and upgrades require redeploying every caller |
| Cloud provider managed model endpoints | Keeping traffic, billing, and residency inside one cloud boundary | You inherit that cloud's model catalogue and its lag behind the model vendors' own APIs |
Treat any hosted gateway as a processor of every prompt your organisation sends. The contract, retention policy, and region of the gateway matter as much as its feature list.