Context Compression — Patterns

Compress from the source, never from the compression

Problem. The cheapest way to keep a session bounded is to summarise the current summary plus the turns since. Each step is faithful to its input, the chain is faithful to nothing, and the divergence cannot be dated because no step was wrong.

Approach. Keep the full original available and regenerate every compression from it, not from the previous output. Where regenerating from the whole source becomes too expensive, fix the segment boundaries so each segment is compressed exactly once from original material, and combine segment outputs by concatenation rather than by re-summarising.

Trade-off. Cost grows with the length of the source rather than staying flat, and the retained original must be stored and governed for as long as the session lives. Concatenating segment summaries also reproduces redundancy across segments and loses the cross-segment narrative that a global rewrite would have given.

Pinned spans passed through verbatim

Problem. The least redundant tokens — identifiers, amounts, dates, names, negations — are exactly the ones a length-reducing rewrite is most likely to normalise, and the result reads perfectly.

Approach. Extract the protected spans before compression, replace them with stable placeholders, compress the remainder, and substitute the originals back. Declare the protected classes per material type rather than globally, and assert after substitution that every placeholder was consumed, failing the compression if the rewrite dropped one.

Trade-off. Placeholders distort the text the compressor sees and can make it less coherent, and the protected set is a maintained list that is always slightly behind reality. Guaranteeing the tokens survive does nothing to guarantee the surrounding claim still applies to them.

Fidelity probes on a held-out question set

Problem. Compression quality is reported as a ratio because the ratio is easy to compute. Nothing in that number says whether the one clause the next turn depended on is still there.

Approach. Build a set of questions answerable from the original material, drawn from real downstream use rather than written by hand. Answer each from the original and from the compressed form and compare. Treat the pass rate as the quality metric and the ratio as the cost metric, and refuse to raise the compression ratio without re-running the probes.

Trade-off. The probe set reflects the questions somebody thought of and is blind to the long tail that compression is most likely to damage. It is also a standing cost that has to be re-run on every change to the compressor, the prompt, or the model behind it.

Compress the tail, freeze the head

Problem. Reducing the oldest material is intuitive and is usually implemented by rewriting the front of the context, which both destroys the task framing and invalidates the cached prefix on every pass.

Approach. Hold the leading region fixed — standing instructions, the task definition, the output contract — and apply compression only to the mutable region behind it. Where history must shrink, replace a contiguous block of middle turns with a marked summary while keeping the recent turns verbatim, so the boundary between compressed and original material is explicit and always in the same place.

Trade-off. The frozen head never shrinks, so a product with a large instruction block and schema has a floor it cannot compress below. Keeping recent turns verbatim is right for coherence and wrong for cost in sessions dominated by long recent messages.