Few-shot Examples — Patterns

Contrastive pairs over additional successes

Problem. Example sets grow by adding another instance of the behaviour that is already working. The model was not confused about the common case; it was confused about the boundary, and nothing in the set shows where the boundary is.

Approach. Add examples in pairs that differ minimally and are handled differently — two inputs that look alike where one warrants an answer and the other a refusal, or where a small change in wording flips the classification. The pair communicates the discriminating feature directly, which a longer list of similar successes cannot.

Trade-off. Minimal pairs are hard to construct and easy to get wrong, and a pair whose difference is not actually the deciding feature teaches a spurious rule with full confidence. They also take two slots to make one point, which is expensive when room is tight.

Synthetic surface, real structure

Problem. The most instructive examples come from production, and shipping production content in every prompt means shipping customer data to every request and into every log that captures prompts.

Approach. Keep the structure, the difficulty, and the phrasing patterns of the real case and replace every identifying element with invented substitutes that are internally consistent — names, account numbers, amounts, dates shifted by a constant. Review the result as a new artefact rather than as a redaction, since the discriminating detail is often the one that was identifying.

Trade-off. Rewriting costs effort per example and can quietly remove the feature that made the case hard, leaving a demonstration of an easier task. Invented values also make the block look artificially tidy, and a model shown only tidy inputs handles messy ones less well.

The block as a versioned set with its own evaluation

Problem. Examples are added individually, each justified by one failing case, and nobody measures what the set does collectively. After a year the block is long, internally inconsistent, and nobody can propose removing anything.

Approach. Treat the example block as one artefact with one version and one owner. Every change — addition, removal, edit — is evaluated against a held-out set that covers the whole task, not only the case that prompted it, and the result is recorded against the version. Removal is then as justifiable as addition, which is the only thing that stops monotonic growth.

Trade-off. It requires a held-out set that stays representative, which is itself a maintained asset, and it slows down the exact situation where people want to add an example quickly. Set-level evaluation also averages over cases, so a change that fixes a rare important case and costs a little on common ones can look like a regression.

Static block with a narrow dynamic tail

Problem. Fully dynamic selection abandons prefix caching and makes every context unique; fully static examples cannot cover a task with many distinct sub-cases.

Approach. Keep the examples that teach format, voice, and the refusal branch in a fixed leading block that never varies, and append at most one or two selected demonstrations behind it for the sub-case at hand. The cacheable region stays intact, the variable part is small, and the two roles are separable when something goes wrong.

Trade-off. It is two mechanisms with two sets of failures, and the selected tail still invalidates everything after it in the cache. Deciding which lessons belong in the fixed block is an ongoing judgement that tends to drift toward making the fixed block larger.