Local and Self-hosted Inference

What it is

Running model weights on hardware you control — a GPU fleet in your own cloud account, an on-premises cluster, or a developer's laptop — instead of calling a hosted endpoint. The API surface may look the same; the engineering does not. You now own accelerator capacity planning, memory arithmetic, batching, model loading, and the full operational lifecycle of a stateful, expensive, latency-sensitive service.

Why it matters

Three motivations drive it, and they lead to different designs. Data residency and confidentiality: prompts never leave a boundary you control. Economics at volume: a saturated accelerator can serve tokens far below hosted per-token prices, provided it stays saturated. Control: open weights do not get deprecated on someone else's schedule, and you can quantise, adapt, or constrain decoding in ways a hosted API does not expose. What all three share is that the hard part is utilisation — an idle GPU costs the same as a busy one.

Key decisions

• Which motivation is primary. Residency tolerates poor utilisation; a cost case does not survive it, and the break-even against hosted pricing is a calculation to do before the hardware is ordered.
• How accelerator memory is divided between weights, activations, and the key-value cache, because the key-value cache is what actually bounds concurrency and is routinely forgotten.
• Whether to quantise, and to what. Quantisation buys memory and throughput and costs accuracy in a way that varies sharply by task and must be measured on yours.
• The batching strategy and the latency it implies, since throughput is bought with queueing and the two cannot be optimised at once.
• The scaling model. Accelerator nodes take minutes to become ready because weights must be loaded, so autoscaling that assumes seconds will always be late.
• The licence terms of the weights, the fine-tunes derived from them, and the outputs — a legal question that belongs in the design, not in a review at the end.

Failure modes

• The deployment is sized by weight size alone; under real concurrency the key-value cache exhausts memory and requests fail or queue unboundedly at a load well below the plan.
• A cost case is built on peak throughput and the fleet runs at low average utilisation, so the effective price per token is several times the hosted alternative it replaced.
• Autoscaling reacts in seconds to a queue that needs minutes of model loading to drain, so it over-provisions through the spike and then scales down into the next one.
• Quantisation is applied for throughput and evaluated only on perplexity or a generic benchmark; the regression lands on long-context reasoning or tool-call formatting, which nothing measured.
• Throughput is tuned by raising batch size until time-to-first-token is unacceptable, while the dashboards show tokens per second improving.
• The serving engine is upgraded for a performance fix and output changes, because kernel and sampler implementations differ between versions even with identical weights.
• A model licence restricting commercial use or output reuse is discovered after the feature ships.

Related capabilities

• Model Fallback — self-hosted capacity is a chain link whose limits you set
• Model Versioning — here a version is weights, quantisation, and engine build together
• Model Gateway — gives self-hosted endpoints the same contract as hosted ones
• Model Benchmarking — the only way to size hardware and validate a quantisation
• Embedding and Reranking Models — the workload most often self-hosted first

Applies: Production readiness · Design questions