Model Gateway — Patterns

Virtual keys over provider credentials

Problem. Provider API keys are long-lived, coarse, and hard to attribute. Handing them to services means every service can call any model at any cost, and revoking one key breaks everything that shares it.

Approach. Issue each caller a gateway-scoped credential that encodes tenant, allowed models, spend ceiling, and expiry. The gateway resolves it to the real provider key, which never leaves the secret store. Revocation, rotation, and re-scoping become gateway operations with no caller change.

Trade-off. The gateway is now an identity system, with the lifecycle work that implies: issuance, expiry, audit, and a recovery path for when the mapping store is unavailable.

Fail-open versus fail-closed policy

Problem. A gateway that must check a quota store, a policy service, and a secret manager before forwarding has inherited their availability. If any check is unreachable, the gateway must either forward unchecked or refuse.

Approach. Classify each check explicitly. Spend ceilings and rate limits can usually fail open against a cached last-known value for a bounded interval. Data-class and tenant-isolation checks fail closed. Write the classification into configuration so the decision is visible rather than emergent from timeout handling.

Trade-off. Fail-open on quota means a dependency outage can be an expensive one, and the bounded interval is a direct cap on how expensive. Fail-closed on policy means a policy service outage is a full product outage.

Streaming passthrough with out-of-band accounting

Problem. Usage totals arrive at the end of a stream, but the gateway must enforce spend limits and emit metrics for a response it has already forwarded chunk by chunk.

Approach. Forward every chunk immediately and accumulate usage alongside the stream. Commit the usage record when the stream terminates, including on client disconnect and on upstream error, where partial output was still generated and billed. Enforcement therefore applies to the next request, not this one.

Trade-off. A single very large request can exceed a ceiling before the ceiling is observed. Pre-flight input-token estimation narrows the window but cannot close it, because output length is not known in advance.

Dual-path deployment to bound blast radius

Problem. A chokepoint concentrates risk exactly where it concentrates control. One bad gateway release stops every AI feature at once.

Approach. Keep a direct-to-provider path that is configuration-switchable per caller, exercised regularly rather than kept as untested dead code, and accept that it runs with reduced enforcement. Pair it with gateway deploys that are canaried on a slice of callers and health-checked on time-to-first-token rather than on process liveness.

Trade-off. The bypass path is a deliberate hole in central enforcement, and it must be treated as one: short-lived, audited, alarmed when used, and never the default.