RAG and Knowledge Engineering
Embedding Generation — Stack
Embedding generation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
What is being chosen here is the serving and orchestration layer for a bulk and an incremental encoding workload, plus the discipline that keeps both sides of the comparison identical.
- Sustained throughput at build scale, measured in units per second at your real unit length, with concurrency, not as a single-item latency figure.
- Rate-limit and quota behaviour. For hosted encoders, the ceiling that determines how long a full re-encode takes, and whether that ceiling can be raised on request.
- Batch determinism. Whether the same text produces the same vector regardless of what else was in its batch, which decides whether a retry can be trusted.
- Revision pinning. Whether the encoder can be addressed by an immutable revision rather than a moving alias, because a hosted model that changes behind a stable name corrupts a corpus without any signal.
- Prefix and pooling control. Whether instruction prefixes, pooling and normalisation are explicit parameters rather than framework defaults that shift between versions.
- Failure granularity. Whether a partial batch failure is reported per item with a stable correlation to the input, rather than as a failed batch.
- Cost model at two scales — the one-off backfill and the steady-state incremental rate — which often favour different deployments.
- Isolation between workloads, so a re-encode cannot consume the capacity the live query path needs for its own encoding.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Hosted embedding APIs with batch endpoints (OpenAI, Cohere, Voyage AI, Google Vertex AI) | Reaching a working corpus with no serving infrastructure | Rate limits set your re-encode duration; confirm the model identifier pins a revision |
| Self-hosted inference servers for encoders (Text Embeddings Inference, vLLM, Infinity, ONNX Runtime) | High-volume backfills where per-unit cost and throughput control matter | GPU capacity planning, batching configuration and pooling defaults all become yours to verify |
| Embedding libraries used in-process (Sentence-Transformers, FastEmbed) | Modest corpora, local development and reproducible fixtures | A library upgrade can change pooling or normalisation defaults and therefore every vector |
| Distributed batch compute for the backfill (Ray Data, Spark, cloud batch jobs) | One-off encoding of very large corpora with checkpointed progress | Correlating outputs back to input identifiers across partitions is where duplication bugs originate |
| Managed pipelines inside a retrieval platform (vendor-managed embedding in a search service) | Small teams who want the encoding step to be invisible | Encoder revision and migration timing are the vendor's decision, not yours |
| Content-addressed embedding cache (object store or key-value store keyed on text hash plus encoder identity) | Corpora with heavy duplication, and repeated reprocessing | The cache key must include every setting that changes the vector, or it serves stale geometry |
Models
Model selection belongs to the model-management side of the estate, and the comparison that justifies a choice — recall on your own judged set, at the candidate depth you actually retrieve — is made there. What belongs here is everything that happens once a model has been chosen. Treat the tuple of model identifier, revision, query prefix, document prefix, pooling strategy, normalisation and output dimension as a single named configuration, store its identity on every vector, and refuse to serve a query whose configuration identity does not match the collection's. When a new model is adopted, the useful question is not whether it is better in general but whether it is better by enough to justify re-encoding the corpus, and that number should be measured on the corpus in question before the migration is scheduled rather than after. Plan the migration against the provider's sustained throughput: a corpus that took four weeks to build the first time will take about four weeks again, and during those weeks the system is serving from one geometry while writing another.