Context Window Management — Patterns

Rollover with an explicit hand-off

Problem. A session reaches the point where the next turn cannot fit. Truncating silently destroys the task definition; refusing ends a conversation the user considers ongoing.

Approach. Define a rollover threshold short of the hard limit and cross it deliberately. Produce a hand-off artefact — the task as currently understood, decisions already made, outstanding obligations, identifiers in play — and open a fresh window seeded with the standing instructions, the hand-off, and the last few turns verbatim. Mark in the conversation record which window each turn belongs to, so a later investigation knows where the discontinuity is.

Trade-off. Everything not written into the hand-off is gone, so the artefact's quality is the session's quality from that point on, and a user reference to "the figure you gave me earlier" may no longer resolve. Rollover also changes the cached prefix, so the turn that crosses the boundary is the most expensive and slowest in the session.

Headroom reserved for a full round-trip

Problem. Occupancy is checked against the limit before the request and fits. The model then issues a tool call, the result comes back, and the follow-up call exceeds the window — leaving the run stuck mid-action, having already caused an effect.

Approach. Make the admission test require room for the entire round-trip: the assembled input, the maximum tool-call output, a realistic upper bound for the result payload, and the final answer. Size the result bound from the observed distribution of that specific tool rather than from a global default, and refuse to begin an action whose round-trip does not fit instead of discovering it halfway through.

Trade-off. Reserving for the worst-case round-trip permanently costs room that most turns never use, and it turns some requests away that would in fact have fitted. Tools with heavy-tailed result sizes force either a generous reserve or a truncation policy on the result itself.

Session occupancy as a monitored distribution

Problem. Request-level token metrics average over short and long sessions and hide the only population that matters: conversations approaching the wall. The problem is discovered from support tickets.

Approach. Track, per session, the high-water occupancy and the number of turns to reach each decile of the window, and alert on the share of active sessions above a threshold rather than on a mean. Record boundary events — rollovers, refusals, evictions — as first-class outcomes with their own counters, so the rate at which sessions hit the limit is a number somebody owns.

Trade-off. Session-scoped metrics need session identity carried through the request path and retained for the session's life, which is state the metrics pipeline would rather not hold. High-cardinality session labels are expensive, so the aggregation has to be done before export and the individual pathological session becomes hard to find afterwards.

Planning against a measured effective length

Problem. The advertised window is an acceptance limit, not a competence limit. Filling it produces requests that succeed and answers that quietly stop using the material in the middle.

Approach. Establish per model build, by probing with task-shaped material placed at varying depths, the occupancy beyond which retrieval accuracy falls below what the product needs. Plan assembly against that number, re-measure when the model build changes, and treat the gap between advertised and measured as a property worth recording next to the limit itself.

Trade-off. The measurement is task-specific and has to be rebuilt for each product and each model release, which is a standing cost. A probe that is easier than the real workload yields a flattering number, and the resulting ceiling is too generous in exactly the cases that matter.