Context Prioritization — Patterns

Rank-space fusion across producers

Problem. Each producer emits a score with its own distribution and its own meaning. Mixing them numerically makes the producer with the most generous scale win, and no amount of weight tuning fixes a comparison between quantities that are not the same kind of thing.

Approach. Rank candidates within each producer, then merge on rank position rather than on raw score, weighting producers by how much the product trusts them rather than by how large their numbers are. Keep the original scores on each candidate for diagnosis, and apply any threshold inside the producer where the score still has meaning.

Trade-off. Rank fusion is blind to margin: a producer that returned one excellent and nine irrelevant candidates contributes its top rank just as strongly as one that returned ten good ones. Products where evidence strength genuinely differs between sources need a calibration step this pattern is designed to avoid.

Floors, caps, and a contested remainder

Problem. A single ranking either outranks something that must always be present, or is patched with so many special cases that the ranking function is no longer the thing deciding.

Approach. Split the room into three parts. Floors are admitted before ranking begins — the current input, the task definition, the output contract — and are not candidates. Caps limit any one producer's share of what remains, so no class can monopolise. Everything left is contested by the fused ranking. Make the floor set small, enumerated, and sized, and fail loudly when floors alone exceed the available room rather than dropping one silently.

Trade-off. Caps forgo optimality by construction: a request where one producer genuinely has all the good material is served worse than an uncapped ranking would serve it. The floor set is also a magnet for additions, and each one permanently reduces the contested remainder.

Position assigned after selection

Problem. Selection and ordering get conflated, so the final sequence is whatever order the selected items happened to be collected in, and the best evidence frequently lands where the model attends to it least.

Approach. Treat the selected set and its arrangement as two steps. Once membership is fixed, lay items out according to a measured policy for the target model — commonly strongest material at the edges of the region rather than its interior, with the user's input last — and keep the layout rule explicit and named so it can be changed and compared. Record the assigned position for each span alongside its score.

Trade-off. An ordering tuned for one model build is a liability after a migration, and re-measuring it is a real evaluation exercise. Grouping by source is also genuinely easier for a model to interpret, so a strictly score-ordered layout can scatter related passages and make the context harder to read.

Shadow ranking before a policy change

Problem. A proposed change to weights or quotas looks better on a hand-built evaluation set that bears little resemblance to the real distribution of candidate sets, and the only way to find out is to ship it.

Approach. Record the full candidate set with its per-producer scores for a sample of real requests. Replay a candidate policy over those recordings and compare what it would have admitted and dropped against what the live policy did — overlap, class mix, and the identity of the newly excluded items. Read the disagreements before changing anything.

Trade-off. Replay shows that the selection changed and cannot show that the answer would have been better, so it narrows risk rather than establishing benefit. Storing complete candidate sets is also substantially more data than storing the chosen context, and it carries the same sensitivity as the material itself.