Context Engineering and Prompt Management

Context Engineering and Prompt Management

Few-shot Examples — References

Few shot examples › References

44 of 57 in this book · 2 min read

The original in-context learning results, the work on what demonstrations actually convey, and the sensitivity findings that make example sets an engineering concern.

Where the capability comes from

  • Language Models are Few-Shot Learners (Brown et al., 2020), arXiv:2005.14165 — the result that a model can be steered by demonstrations placed in the context with no weight updates, including the measured effect of varying the number of demonstrations.
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022), arXiv:2201.11903 — demonstrations that show intermediate working rather than only the answer, and the case where the content of an example matters more than its format.

What demonstrations actually convey

  • Min and colleagues, 2022, on what makes in-context learning work (EMNLP 2022) — the finding that the input distribution, the label space, and the format of demonstrations carry much of the effect, and that correct pairing matters less than expected for some tasks. The strongest caution against assuming an example teaches what its author intended. Cited by name.
  • Zhao and colleagues, 2021, on calibrating before use to improve few-shot performance (ICML
    1. — the measured bias introduced by the examples' label distribution and ordering, and why a classification prior can be an artefact of the block rather than of the input. Cited by name.

Order and selection

  • Lu and colleagues, 2022, on few-shot prompt order sensitivity (ACL 2022) — that permuting the same demonstrations changes measured accuracy substantially, which makes example order a pinned decision rather than an incidental one. Cited by name.
  • Liu and colleagues, 2022, on selecting good in-context examples by nearest-neighbour retrieval — the origin of similarity-based dynamic selection and its measured gains over a random block. Cited by name.
  • Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych, 2019), arXiv:1908.10084 — the bi-encoder formulation behind the embedding step that dynamic selection depends on.

No verified reference is recorded for contrastive example pairs, for synthesising examples from production cases, or for evaluating an example block as a versioned unit. These are practices built on the results above rather than published findings of their own.