Context Prioritization

What it is

Deciding, among candidates competing for a fixed amount of room, which ones are present in the final context and in what position. The candidates are heterogeneous by nature — a retrieved passage, a remembered fact, a tool result, a prior turn, a worked example — and they arrive carrying scores that were computed by different mechanisms on different scales and mean different things. Prioritisation is mostly the work of making those candidates comparable, and then of treating position as a decision rather than as a by-product of the order things were appended in.

Why it matters

The room available is smaller than the material that could plausibly help, on nearly every non-trivial request. What gets admitted determines the ceiling on answer quality, and no downstream capability can recover from a passage that was never shown. Position matters almost as much: the same passage is used differently depending on where in a long context it sits, so a ranking that gets inclusion right and ordering wrong has done half the job. And because the comparison is across incommensurable sources, the default behaviour — whichever producer happens to emit the largest numbers wins every slot — is both easy to implement and badly wrong.

Key decisions

• How scores from different producers are made comparable. Calibrating each to a shared meaning is principled and requires labelled data; combining by rank rather than by score avoids the calibration problem and discards the margin between a strong and a weak match.
• Whether admission is one global ranking or a set of per-class allocations. A single ranking lets the best evidence win regardless of origin and lets one prolific producer take every slot; fixed allocations guarantee diversity and will reserve room for a class that had nothing useful this time.
• What is unconditionally present. The current user input, the active task definition, and the output contract are not candidates; naming them explicitly as floors is what stops a ranking function from ever outranking them.
• Whether redundancy is penalised. Ranking purely by relevance fills the available room with five phrasings of the same fact, because each of them is individually relevant.
• How position is assigned once inclusion is settled — strongest first, strongest last, or grouped by source — and whether that choice has been measured on this product's own traffic rather than inherited.
• Whether the rule is deterministic, including tie-breaking. Two candidates with equal scores resolved by dictionary iteration order produce an ordering that changes between runs and between library versions.
• Whether drops are recorded. A ranking that cannot tell you what it rejected cannot be debugged and cannot be compared against its replacement.

Failure modes

• A retriever's similarity scores and a recency score both occupy the zero-to-one range but concentrate in different parts of it, so one source wins essentially every contest and the other is effectively disabled without anyone deciding to disable it.
• Relevance is computed against the literal last message. The user asks "and the second one?", the scorer has almost nothing to match on, and the request proceeds with the weakest candidate set of the whole conversation.
• Pinned material accumulates. Every incident adds a must-include, and eventually the pinned set alone exceeds the available room, at which point the pinning mechanism silently starts dropping pins.
• The highest-scoring passage is placed in the middle because the assembler groups by source and that source sits in the middle, so the best evidence lands in the least-used position.
• Ranking is tuned on an offline set where every item came from one producer, then deployed over a mixed candidate stream it has never seen.
• One class dominates by construction — chunking produces hundreds of document fragments and memory produces four facts — so the global ranking is a document ranking with a rounding error of memory in it.
• The scoring weights are changed and nothing records the previous ordering for past requests, so a complaint about last week's answer cannot be reproduced even with the same inputs.
• Deduplication happens after ranking rather than before, so three copies of the same passage consume three of the ten slots and are then collapsed into one.

Related capabilities

• Context Window Management — supplies the room this capability allocates
• Context Compression — shrinks an admitted item rather than choosing between items
• Dynamic Prompt Assembly — gathers the candidate set and applies the resulting order
• Few-shot Examples — a candidate class whose selection has its own criteria
• Context Provenance — the source metadata that makes class-aware allocation possible

Applies: Production readiness · Design questions