Model Management and Inference
Embedding and Reranking Models — Stack
Embedding and reranking models › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
- Retrieval quality on your corpus. Measured with your own queries and judgements. Public averages are a shortlist mechanism and nothing more.
- Maximum input length versus your chunk size. Including how the implementation behaves past the limit: truncate silently, error, or pool over windows.
- Symmetric or asymmetric usage. Whether the model expects distinct query and document instructions, and whether your pipeline applies them consistently everywhere.
- Dimensionality and truncatability. Storage and query cost scale with dimension; models trained for dimension truncation let you choose the point later.
- Throughput for the initial build. Embedding a large corpus is a bulk job measured in passages per second per device, and it is often the deciding practical constraint.
- Reranker latency as a function of candidate depth. Measure at the depth you intend to use, under concurrency, not at depth one.
- Language and domain coverage weighted by your actual traffic distribution, not by the number of languages listed.
- Licence and deployability for open-weight options, including whether the weights may be used commercially and whether derivatives may be distributed.
- Stability of the hosted endpoint, for API models: an embedding model that changes behind a stable name would silently corrupt an index.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Hosted embedding APIs (OpenAI, Cohere, Voyage AI, Google) | Getting to a working index quickly with no serving infrastructure | Per-token cost on the full corpus build and on every query; confirm the model identifier is version-pinned |
| Open-weight embedding families (E5, BGE, GTE, Nomic Embed, Jina Embeddings) | Self-hosting, domain fine-tuning, and avoiding per-query cost at volume | Many require specific query/document instruction prefixes; read the model card before benchmarking |
| Hosted reranking APIs (Cohere Rerank, Voyage rerank, Jina Reranker) | Adding a precision stage without running a cross-encoder yourself | Latency includes a network round trip on the critical path, multiplied by nothing — but the round trip is per request |
| Open-weight cross-encoder rerankers (BGE reranker, MiniLM cross-encoders, Qwen-based rerankers) | Latency control and no per-query cost, where a GPU is already available | Compute scales with candidate depth times passage length; cap the depth explicitly |
| Late-interaction retrieval (ColBERT-style multi-vector models) | Strong recall without a separate reranking stage | Index size grows substantially; needs a store that supports multi-vector scoring |
| Sparse and learned-sparse retrieval (BM25, SPLADE) as a hybrid partner | Exact terms, identifiers, and rare tokens that dense vectors handle poorly | Hybrid scoring needs a fusion rule, and the fusion weights are another thing to tune and maintain |
| Serving frameworks (Sentence-Transformers, Text Embeddings Inference, ONNX Runtime) | Running open-weight encoders efficiently, including on CPU | Pooling and normalisation defaults differ between frameworks; a mismatch changes every vector |
Models
This is the capability where model choice is the capability, so treat the selection as an engineering decision with recorded evidence. Build a judged set from your own corpus — queries with known relevant passages — and compare candidates on recall at the candidate depth you intend to retrieve, then on ranking quality after reranking. Hold chunking, instructions, and normalisation fixed while the model varies, or the comparison measures your pipeline rather than the model. Expect a general-purpose open-weight embedding model of modest size to be competitive with a hosted one on ordinary prose, and expect the gap to appear on specialised vocabulary, where a domain-adapted or fine-tuned model usually wins. Choose dimensionality after the quality comparison, not before, and record the model identifier, revision, instruction prefixes, pooling strategy, and normalisation as a single unit — every one of them changes the vectors, and the index is only valid for the exact combination that produced it.