Token Budget Management

What it is

Deciding, before a call is made, how many tokens a unit of work is allowed to consume, and enforcing it. The unit may be a request, a conversation, an agent run, a tenant's month, or a background job. It covers input assembly under a ceiling, reserving room for the output, accounting for tokens the model consumes invisibly while reasoning, and defining what happens when the ceiling is reached.

Why it matters

Two hard limits sit behind every call and they are not the same limit. The context window is a physical ceiling: exceed it and the request is rejected or silently truncated. The spend ceiling is a commercial one: exceed it and the product is unprofitable or the quota runs out mid-month. Agentic workloads make both acute, because a loop that retries, calls tools, and re-reads its own history can consume more tokens in one run than a thousand ordinary requests, and the growth is not visible from any single call.

Key decisions

• What the budgeted unit is. Per-request ceilings are easy and miss runaway loops; per-run and per-session ceilings catch them and require carrying state across calls.
• How much of the window is reserved for output before input assembly begins. Filling the context with retrieved material and discovering there is no room left to answer is the characteristic mistake.
• What is dropped first when the input does not fit: the oldest turns, the lowest-scoring retrieved passages, the least recent tool results, or whole optional sections. The order is a product decision, not a technical one.
• Whether the count is estimated or exact. Exact counting needs the target model's tokenizer; estimates are cheap and are wrong by enough to matter near a ceiling.
• The behaviour at the ceiling: truncate, summarise and continue, degrade to a smaller context, refuse with an explanation, or queue for later. Each is defensible; silently truncating is not.
• Whether reasoning tokens, cached input tokens, and tool-result tokens are priced and counted separately, since they are billed differently and often omitted from estimates.

Failure modes

• Retrieved context is packed to fill the window, leaving too little output allowance; the answer is cut off mid-sentence and the cause looks like a model problem.
• A character-count heuristic approximates tokens; it holds for English prose and underestimates badly for code, non-Latin scripts, and long identifier-heavy JSON, so ceilings are exceeded exactly on the inputs that were already hardest.
• Conversation history grows unbounded. Cost per turn climbs linearly, the earliest turns stop mattering, and quality degrades as useful material is crowded out of the middle.
• An agent loop has a step limit but no token limit, so a single pathological run costs more than the day's normal traffic before the step limit stops it.
• Budgets are enforced per request but attributed per tenant only in a nightly job, so a tenant exhausts a monthly allowance in an afternoon and the system learns the next day.
• Extended-reasoning tokens are generated and billed but not counted by logic that only looks at visible output, making measured cost per request reliably too low.

Related capabilities

• Inference Configuration — the output ceiling is set there and consumed here
• Model Capability Registry — supplies context limits, tokenizer family, and prices
• Model Gateway — the enforcement point when budgets must bind across services
• Semantic Caching — a hit removes the spend entirely rather than reducing it
• Multi-model Support — token counts are not comparable across model families

Applies: Production readiness · Design questions