Context Compression

What it is

Replacing a span of material with a shorter span intended to serve the same purpose: summarising a stretch of dialogue, reducing a verbose tool payload to the fields that were asked about, collapsing near-identical passages into one, or rewriting prose into a denser notation. It is a lossy transformation applied to material that has already earned a place, and its defining question is not how much smaller the result is but which facts survived.

Why it is not prioritisation

Prioritisation decides whether an item is present at all and where it sits; compression changes what an already-admitted item looks like. The two fail in opposite ways and are diagnosed differently. A prioritisation failure leaves a visible hole — the relevant document is simply absent, and the model says it does not know. A compression failure leaves no hole at all: the item is present, reads fluently, and no longer contains the clause, the number, or the negation that mattered, so the model answers confidently from a text that has quietly changed meaning. Selection errors are found by asking what is missing; compression errors are only found by comparing against the original.

Why it matters

Occupancy grows and conversations do not get shorter, so any long-lived assistant eventually has to re-express its history rather than carry it. Done well, compression is what lets a hundred-turn session stay coherent and affordable. Done carelessly it is the most dangerous transformation in the pipeline, because it introduces errors that are indistinguishable from the model's own and attributable to nothing: the summary of the summary of turn four asserts a decision that was never made, and every subsequent turn builds on it.

Key decisions

• Whether compression is extractive or abstractive. Selecting original sentences preserves wording and cannot invent; rewriting reads better, compresses far harder, and can assert things the source did not.
• Whether the original is retained and addressable after compression. If it is, compression is an optimisation that can be undone on demand; if it is not, it is a destructive edit to the system's only record.
• What is declared never-compressible: identifiers, quantities, dates, legal or policy text, the user's own stated requirements. Without an explicit list these are compressed first, because they are the least redundant parts of the text.
• Where compression runs. Compressing at ingest amortises the cost across reuse and may discard something a later request needed; compressing at assembly time is request-aware and pays the latency on the hot path.
• Whether compressed output is ever recompressed. Repeated passes drift: each generation is faithful to the last and the chain is faithful to nothing, and the drift is invisible because every step looked reasonable.
• How fidelity is measured at all. A length ratio is trivially measurable and says nothing; verifying that the specific facts a downstream task needs survived requires a question set and a comparison against the source.
• Whether the compressed artefact is marked as compressed, so downstream readers know they are looking at a derivative rather than at what was said.

Failure modes

• A rolling summary is regenerated from the previous summary every few turns. By turn sixty the context contains a confident narrative that diverges from the transcript in ways nobody can date.
• The summariser drops a negation. "The customer did not approve the refund" becomes "the customer approved the refund", and every later turn is correct reasoning over a false premise.
• Numbers and identifiers are paraphrased — an order number loses digits, a currency amount is rounded, a version string is normalised — and the agent then acts on them.
• Compression is applied to the user's own most recent instruction because it was long, so the model is answering a paraphrase of the request rather than the request.
• The compression step is itself a model call on the critical path, and the latency saved by sending fewer tokens is smaller than the latency spent producing them.
• A summary is cached and reused across requests with different needs; it was written to answer one question and silently omits what the next request required.
• The compressed form is stored in the same field as original content with no marker, so later provenance analysis attributes an invented sentence to a source document.
• Compressing the leading region of the context rewrites the stable prefix every few turns, which defeats prefix caching and costs more than the compression saved.

Related capabilities

• Context Window Management — the occupancy pressure that makes this necessary
• Context Prioritization — decides what is present; this decides how much of it
• Context Provenance — what must survive a rewrite if a span is to stay attributable
• Memory Consolidation — the equivalent reduction applied to stored memory rather than to a request
• Dynamic Prompt Assembly — where request-time compression is invoked

Applies: Production readiness · Design questions