Embedding and Reranking Models

What it is

The two small models that sit on either side of a retrieval step. An embedding model maps text to a vector so that related passages end up near each other, which is what makes approximate nearest-neighbour search possible at corpus scale. A reranker — typically a cross-encoder — reads a query and a candidate passage together and scores their relevance directly, which is far more accurate and far too slow to run over a whole corpus. They are a pair: the embedding model recalls a wide candidate set cheaply, the reranker orders the top of it precisely.

Why it matters

Retrieval quality caps generation quality. A generator cannot cite a passage that was never retrieved, and no prompt engineering recovers from a candidate set that does not contain the answer. These two models decide what the generator is allowed to see. They also carry a migration cost unlike anything else in the estate: changing the embedding model invalidates every stored vector, so the choice is far stickier than a generation-model choice and deserves more care up front.

Key decisions

• Dimensionality, traded against index memory, query latency, and storage. Models offering truncatable representations let one index serve several operating points, provided the truncation is applied identically at write and query time.
• Maximum sequence length, and whether it fits your chunking. A model that truncates at a limit shorter than your chunks will silently drop the tail of every long passage.
• Whether the model is asymmetric — expecting a distinct instruction or prefix for queries and for documents. Getting this wrong degrades recall quietly rather than failing.
• The distance metric and whether vectors are normalised, since the index, the model, and any cached threshold must agree on one answer.
• Whether a reranking stage exists at all, and if so, the candidate depth it reads. Depth is the dial that trades recall against latency and cost.
• Whether to use a general model, a domain-adapted one, or to fine-tune on your own labelled pairs — and whether you have, or can obtain, the labels that would justify the last option.
• Hosted or self-hosted: these models are small enough that self-hosting is genuinely practical, which is less often true elsewhere.

Failure modes

• The embedding model is upgraded and only new documents are re-embedded, so the index contains two incompatible geometries and similarity scores become meaningless.
• Queries are embedded with a different prefix or instruction than documents were, and recall drops by a wide margin with no error anywhere.
• Chunks exceed the model's maximum sequence length and are truncated at encode time; the content past the limit is unretrievable and nothing reports it.
• A reranker is added over a candidate set that did not contain the answer. Ordering improves, retrieval does not, and the latency is spent for nothing.
• Reranking depth is raised to improve recall and p95 latency degrades linearly with it, because a cross-encoder scores every pair rather than reusing a cached vector.
• A model is chosen on a public leaderboard average and underperforms on the corpus that matters, because the benchmark's domains do not resemble yours.
• Truncated embedding dimensions are used for the query and full dimensions for the documents, or vice versa, producing silently wrong distances.
• A multilingual model is adopted for coverage and is worse in the one language that carries most of the traffic.

Related capabilities

• Semantic Caching — depends on an embedding model, but needs one that separates rather than groups
• Local and Self-hosted Inference — where these small models are most often run in-house
• Model Benchmarking — supplies the domain-specific evaluation this choice should rest on
• Model Versioning — an embedding model version is part of the index's identity
• Model Capability Registry — records dimensionality, sequence limits, and asymmetry

Applies: Production readiness · Design questions