Model Gateway

What it is

A network service that every model call passes through on its way out of the organisation. It holds the provider credentials, applies quota and policy, emits the telemetry, and returns a response the caller could not have obtained directly. Where Multi-model Support is an abstraction and Model Routing is a decision, the gateway is a deployed process with an address, an on-call rotation, and a latency budget of its own.

Why it matters

Without a chokepoint, provider API keys spread across services and notebooks, nobody can answer which team spent the quarter's inference budget, and a policy change — block this data class, cut this tenant off, move this workload to another region — has to be implemented separately in every repository that calls a model. The gateway exists so that those become configuration changes in one place rather than a migration.

Key decisions

• Whether it is a separate network hop or a library linked into each service. A hop buys central enforcement and costs a round trip, a deployment, and a new thing that can be down.
• What the gateway is authoritative for and what it merely observes. Credential custody and quota enforcement belong to it; prompt construction generally does not.
• How callers authenticate to it, and whether the caller's identity — service, tenant, end user — survives as far as the usage record and the per-tenant quota counter.
• Whether responses are proxied as a stream or buffered. Buffering simplifies inspection and policy, and destroys time-to-first-token for interactive products.
• How much request and response content is retained, for how long, and who can read it, given that the gateway sees every prompt in the organisation.

Failure modes

• The gateway becomes the outage: a deploy, a connection-pool exhaustion, or a dependency it added takes down every AI feature simultaneously, including the ones with no provider problem at all.
• Streaming is implemented as buffer-then-forward, so first-token latency equals full generation time and interactive products feel broken while every metric reads healthy.
• Long-lived upstream connections are cut by an idle timeout in a load balancer that was never configured for multi-minute responses; callers see truncation, not an error.
• One tenant's burst consumes the shared upstream rate limit and every other tenant is throttled, because quota is tracked in aggregate rather than per caller.
• Request bodies are logged verbatim for debugging and the log store silently becomes the largest concentration of customer content in the estate.

Related capabilities

• Multi-model Support — the abstraction the gateway usually exposes
• Model Routing — the decision the gateway is the natural place to execute
• Token Budget Management — quota the gateway is positioned to enforce
• Semantic Caching — a cache placed at the gateway serves every caller at once

Applies: Production readiness · Design questions