Context Engineering and Prompt Management
Context Compression — Stack
Context compression › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
Every candidate trades length against fidelity; what distinguishes them is whether the loss is bounded, measurable, and reversible.
- Loss profile. Whether the mechanism can only remove content or can also assert content the source did not contain. That single property separates safe from dangerous.
- Determinism. Whether the same input compresses to the same output. Non-deterministic compression makes two otherwise identical requests incomparable and makes replay impossible.
- Ratio control. Whether a target length can be requested and is actually honoured, or whether output length is whatever the mechanism produces.
- Preservation guarantees. Whether entities, numbers, and quoted spans can be pinned so they pass through unaltered.
- Added latency and cost. Measured against the saving. A compressor that costs a model call to remove tokens from a model call has to earn its place arithmetically.
- Reversibility. Whether the original remains addressable, so a downstream step can fetch the full text when the compressed form proves insufficient.
- Evaluability. Whether the mechanism produces anything a fidelity check can attach to — alignment between output spans and source spans, or a per-fact trace.
- Idempotence. What happens when the output is fed back in, since every long-running session eventually does this.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Deterministic structural reduction (field projection on JSON, head-and-tail truncation, column selection on tabular results) | Verbose tool payloads with a known shape, which are usually the largest avoidable occupant | Needs a per-tool decision about what matters; a projection written for one API version silently drops a new field |
| Extractive selection (sentence scoring, TextRank-style ranking, query-conditioned sentence retention) | Reducing documents while guaranteeing every retained word appeared in the source | Reads badly, keeps redundancy between retained sentences, and loses the connective tissue that made a passage coherent |
| Token-level prompt compressors (LLMLingua and its long-context variants, selective-context methods) | High compression on long prompts where a small model can score token importance | Output is not human-readable, so review is impossible; effectiveness varies sharply by task and must be measured per use |
| Near-duplicate collapsing (MinHash, SimHash, embedding-similarity clustering) | Multi-source assembly where the same passage arrives several times | Collapses passages that differ only in a number or a negation; threshold tuning is the whole job |
| Abstractive summarisation by a model call | Conversation history and long narrative material where structure cannot be exploited | Can invent; costs latency and tokens; non-deterministic unless sampling is pinned, and even then not across model builds |
| Rolling or hierarchical summaries in agent frameworks (summarising buffers, summary nodes, OS-style paging designs) | Long-running assistants that must continue indefinitely | Recompression drift is the default behaviour; most implementations summarise the summary unless explicitly told not to |
| Retrieval over the full text instead of compression | Cases where the material is addressable and only a fraction is ever needed per turn | Shifts the problem to retrieval quality, adds a dependency to the assembly path, and does not help with dialogue history |
Models
Abstractive compression is a model choice, and it is usually a different model from the one answering.
- Small instruction-tuned models are the common default for turn-level summarisation. They are cheap enough to run every few turns and weak enough that a tightly constrained prompt and a narrow input window matter more than usual.
- The answering model itself gives the best fidelity and the worst economics, and makes compression latency scale with the same capacity constraint as the product's main path.
- Fine-tuned or distilled summarisers pay off when the material has a fixed shape — support transcripts, meeting notes, one API's payloads — and are the only option that reliably honours a target length.
- Long-context cheap models suit the one-shot reduction of a very large document, where the alternative is chunk-and-merge with its own seams and contradictions.
- Whichever is chosen, pin the build. A summariser upgrade changes the compressed form of every future session and is indistinguishable, in the metrics, from a change in the answering model.