Embedding Generation

What it is

The production stage that converts every indexed unit into a vector and keeps the correspondence between the two exactly right — the bulk job that encodes a corpus, the incremental path that encodes what changed, the matching query-side encoding, and the record of which encoder, revision, instruction prefix, pooling rule and normalisation produced each stored vector. Choosing which encoder to use is a different decision, made elsewhere; this is the machinery that applies it consistently to tens of millions of units and survives the day it changes.

Why it matters

A vector store has no way to tell that two of its vectors were produced by different models or with different conventions. Distances between them are still computed, still ranked, and still returned, so a mismatch does not fail — it degrades, uniformly and silently, across whatever fraction of the corpus is affected. The same property makes an encoder change the single most expensive operation in a retrieval system: it invalidates every stored vector at once, and a corpus whose re-encoding takes six weeks at the provider's rate limit needs that plan written before the upgrade, not after. Throughput at build time is often the binding constraint on how large a corpus a product can have at all.

Key decisions

• Where the encoding provenance lives. Model identity, revision, prefix, pooling, normalisation and dimension are one indivisible fact about a vector; recording them on the index rather than on the vector makes a partially re-encoded corpus undetectable.
• The migration shape when the encoder changes: a parallel collection built alongside the live one and swapped, dual-writing into both for a comparison period, or in-place replacement that leaves the corpus mixed for the duration.
• How the query path is bound to the document path. The safest designs make it impossible to encode a query with different settings than the corpus, because the alternative is a configuration that must be kept in sync by convention.
• Batching and concurrency against the provider's or the device's limits, and whether the batch size affects the result — it does, wherever padding or truncation behaviour varies by batch.
• Deduplication before encoding. Corpora contain large volumes of identical text, and a content-addressed cache turns a substantial fraction of a backfill into lookups.
• What happens to a unit whose encoding failed: retried, quarantined, or counted as done. The third produces a corpus with holes that only an index-to-source reconciliation reveals.
• Whether a reduced dimension is used, and the guarantee that reduction is applied identically on both sides of every comparison.

Failure modes

• A rolling deployment lands mid-backfill, the second half of the corpus is encoded by a different model revision, and the index now contains two incompatible geometries with no field recording which is which.
• The query path is upgraded to a new encoder before the corpus is re-encoded, and retrieval quality collapses to near-random while every service reports healthy.
• An asymmetric model's document prefix is applied during the build and omitted at query time, and recall drops by a wide margin with no error raised anywhere.
• Units are encoded in batches and the vectors are matched back to units by position in the response array, so a provider that drops or reorders a failed item shifts every subsequent assignment by one.
• A transient provider failure is retried with a smaller batch, different truncation applies, and a subset of the corpus is encoded from truncated text.
• Vectors are normalised by one framework's defaults and not by another's after a library upgrade, so all distances computed against the newer portion are on a different scale.
• Three percent of units fail permanently, the failures are logged and not counted, and the documents they came from are unretrievable without any report saying so.
• A re-encode is started without reserving capacity, saturates the shared encoder quota, and the incremental path stalls for the entire duration of the migration.

Related capabilities

• Embedding and Reranking Models — where the encoder itself is chosen and benchmarked
• Chunking Strategies — produces the units this stage encodes, and is bounded by its sequence limit
• Vector Database — the store that accepts these vectors and must agree on dimension and metric
• Incremental Indexing — determines how much of the corpus this stage has to touch per update
• Knowledge Versioning — the encoder revision is part of a corpus state's identity

Applies: Production readiness · Design questions