Context Window Management

What it is

Managing the occupancy of a finite, physically bounded window across the life of a conversation or an agent run: what sits inside it on each call, how quickly it fills as turns and tool round-trips accumulate, how much of it is genuinely usable, and what the product does at the point where the next turn will not fit. The window is a per-call capacity shared by everything the model must read and everything it is about to write, and unlike almost every other constraint in the system it cannot be relieved by provisioning more of anything.

Why it is not a token budget

A budget is an allowance a unit of work is permitted to spend, chosen by the organisation and enforced because spending has a price; it can be raised by a decision, and it spans many calls. The window is a property of the model being called: it is the same for every tenant, it cannot be raised, and it binds one call at a time. The two interact — a run that stays inside its window every turn can still blow its allowance over ninety turns — but they fail differently. Exceeding a budget produces a policy decision; exceeding a window produces a rejected request or a silently truncated one. Allowances, their enforcement points, and per-run accounting belong to Token Budget Management.

Why it matters

Occupancy only ever rises within a session. Every turn appends, every tool call appends twice, and every retry appends again, so a conversation that works perfectly for twenty exchanges breaks at the twenty-first — and it breaks for the users with the longest, most valuable sessions. The advertised window is also not the usable one: attention to material in the middle of a long context degrades, and providers charge framing overhead the application never counted. A product that plans against the number in the model's datasheet has planned against a number it will not get.

Key decisions

• Where the headroom line sits, expressed in the target model's own tokenizer rather than in characters, and whether it leaves room for one full tool round-trip rather than only for the next message.
• What happens at the boundary: refuse the turn with an explanation, roll over into a fresh window seeded with a hand-off, or evict from the existing one. Each is a different, visible product event, and choosing none of them means the provider chooses truncation.
• Whether occupancy is instrumented per session rather than per request, since a request-level metric cannot tell you how many live conversations are two turns from the wall.
• Whether a window is reset at task boundaries, and who is entitled to declare a task finished — the user, the orchestrator, or a heuristic over the last few turns.
• Whether the leading portion of the occupancy is held byte-identical between turns. Anything variable placed in front invalidates every cached prefix and changes the cost and latency curve of the whole product.
• Whether planning uses the advertised limit or a smaller measured one established by probing how well the model actually retrieves from depth.
• How the limit itself is obtained. A constant in the code is wrong the week a model is replaced; reading it from a maintained source makes the model's own metadata a dependency.

Failure modes

• The limit lives as a literal in one file. A model is swapped for a cheaper variant with a smaller window and every long-running session breaks within the hour, all at once.
• Occupancy is measured only on successful requests, so the distribution looks comfortable and every session that hit the wall is missing from the data that would have shown the problem.
• Growth is dominated by verbose tool results nobody counted; three retries of one noisy call consume a quarter of the window and the conversation becomes unusable immediately after a specific action, which is the hardest kind of bug to reproduce.
• The eviction rule is "drop the oldest messages", applied to a window whose first message carries the task definition, so the agent forgets what it was asked to do while remembering the details of how it failed.
• A current timestamp, a turn counter, or a shuffled example set is placed near the front of the window on every turn, so nothing before the user's message is ever reusable and the saving the team believed it was getting never existed.
• Fixed occupants are forgotten in planning: the output schema, the tool declarations, and the example block are present on every call and together are frequently larger than the conversation they precede.
• Occupancy is checked against the limit before each call and never projected forward, so nothing says that a session growing by three thousand tokens a turn has two turns left. The wall arrives as a failed request rather than as a transition the product had time to plan.
• Long-context support is enabled and quality falls, because the material that matters is now in the middle of a very large window and the extra room was spent on passages nobody needed.

Related capabilities

• Context Compression — the way occupancy is reduced without dropping material outright
• Context Prioritization — decides what gets the room this capability measures
• Dynamic Prompt Assembly — the component that must respect the ceiling at request time
• Conversation Memory — the record a rolled-over window is derived from
• Token Budget Management — the commercial allowance, as distinct from the physical limit

Applies: Production readiness · Design questions