Token Budget Management — Patterns

Reserve output before assembling input

Problem. Context assembly treats the window as space to fill. Retrieved passages, history, and tool results expand to occupy it, and the generation then has no room — or the request is rejected outright for exceeding the window.

Approach. Compute the assembly ceiling as the context window minus the configured maximum output, minus any reasoning allowance, minus a safety margin for counting error and for the provider's own framing overhead. Assembly fills only the remainder, and the reservation is subtracted first rather than checked last.

Trade-off. The reservation is unused whenever the answer is short, so a generous output allowance permanently costs context that retrieval could have used. Tuning it is a direct trade between how much evidence the model sees and how long an answer it can give.

Priority-ordered context admission

Problem. When the assembled input does not fit, something must go. Truncating from one end is arbitrary and routinely discards the most important material, because importance is not correlated with position.

Approach. Assign every candidate block a priority and a minimum viable form before assembly: the current user turn is mandatory, the system instructions are mandatory, retrieved passages are ranked and droppable below a score, old turns are summarisable, tool results are truncatable to a head and tail. Admit in priority order until the ceiling is reached, and record what was dropped on the request.

Trade-off. Priorities are a product judgement that becomes a maintained configuration, and recording drops is only useful if someone reviews them. The scheme also makes behaviour input-dependent in a way that complicates reproducing a specific past answer.

Per-run budget carried through an agent loop

Problem. Each individual call in an agent run is within its per-request limit. The run makes ninety of them. No single call was unreasonable and the total is.

Approach. Allocate a token budget to the run as a whole, carry the remaining balance in the run's state, and debit actual usage after every call including failed attempts. Before each step, require that the remaining balance covers a realistic worst case for that step; if it does not, terminate the run with a partial result and an explicit exhausted-budget outcome rather than a generic error.

Trade-off. Runs now fail for reasons unrelated to the task, and users need a way to understand and raise the limit. Debiting failed attempts is correct — they were billed — but it means an unlucky run gets less work done for the same money.

Reserve-then-reconcile for shared ceilings

Problem. Output length is unknown before generation, so a ceiling checked only on input size cannot bind actual spend, and a ceiling checked after the response has already been exceeded.

Approach. Before the call, reserve the worst case — counted input plus configured maximum output plus reasoning allowance — against the tenant's remaining balance, and reject if the reservation does not fit. After the call, reconcile the reservation against the reported usage and release the difference, including on error and on client disconnect where partial output was still generated.

Trade-off. Reserving the worst case makes the ceiling conservative: tenants are refused while nominally in budget, because most responses are far shorter than the maximum. Shrinking the reservation using an observed length distribution restores throughput and reintroduces the possibility of overshoot.