Embedding Generation — Patterns

Encoding configuration as a named, stored identity

Problem. A vector is only comparable to vectors produced by exactly the same settings, and there are more settings than anyone remembers: model, revision, prefix on each side, pooling, normalisation, truncation, dimension. Any one of them drifting produces silent degradation rather than an error.

Approach. Give the whole tuple a single identifier, computed as a hash of its contents, and store that identifier on every vector and on the collection. The query path resolves its own configuration identifier and compares it with the collection's before issuing a search, failing loudly on a mismatch. A partial re-encode then shows up as a collection containing two identifiers, which is a queryable fact rather than a theory.

Trade-off. The identifier must cover every input that affects the output, including library versions and tokeniser revisions, and anything omitted produces false confidence. It also makes legitimate experiments heavier, because trying a different prefix means a new identity rather than a configuration tweak.

Parallel collection build with a verified cut-over

Problem. Changing encoders invalidates every stored vector, and the corpus cannot be unavailable for the days or weeks the re-encode takes.

Approach. Build the new geometry into a separate collection while the existing one serves, feed the incremental stream into both, and hold the cut-over behind a verification step: unit counts reconciled against the source of truth, and a judged query set evaluated against both collections so the change is quantified rather than assumed. Switch an alias, keep the old collection until the comparison has survived real traffic, then reclaim it.

Trade-off. Storage and encoding spend roughly double for the migration window, and the dual-write path is a second place where the two collections can diverge. Reverting is only genuinely available while the old collection exists, so the reclamation date is a risk decision rather than a cleanup chore.

Content-addressed encode cache

Problem. Real corpora repeat themselves — boilerplate clauses, shared templates, duplicated pages across sources — and a reprocess triggered by an unrelated configuration change re-encodes text whose vector is already known and still valid.

Approach. Key a cache on the hash of the normalised unit text combined with the encoding configuration identity, and consult it before dispatching any unit to an encoder. A backfill then costs only the genuinely novel text, and a change to a downstream stage costs nothing here at all.

Trade-off. The cache is another store to size, expire and keep consistent, and its hit rate collapses to zero the moment the configuration identity changes, which is exactly when the backfill is largest. Normalisation before hashing is also a correctness decision: too aggressive and distinct text collides, too lax and the hit rate disappears.

Reconciliation between source units and stored vectors

Problem. Encoding failures are counted in logs, not in the corpus, so a few percent of units can be missing from the index indefinitely while every dashboard is green. Absence does not raise alerts; it answers questions slightly worse.

Approach. Run a periodic set-difference between the units the pipeline believes exist and the vector identifiers the store actually holds, in both directions. Units with no vector are re-enqueued; vectors with no unit are candidates for removal. Report the two counts as standing metrics rather than as the output of an investigation.

Trade-off. The comparison is expensive at corpus scale and usually has to be sampled or partitioned, which means it detects systematic gaps quickly and individual ones slowly. Orphan vectors also need care: a vector with no unit may be a deletion that has not propagated rather than an error, and removing it eagerly can mask a real defect elsewhere.