RAG and Knowledge Engineering
Knowledge Versioning — Stack
Knowledge versioning › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The requirement decomposes into recording identity, retaining content, and serving a query against something other than the present. Each has a different cost profile.
- Granularity of what can be versioned — the document, the extracted text, the unit, the index generation, or the whole pipeline configuration as one object.
- Point-in-time query support. Whether the store can answer as of a past state, or whether the only way to query the past is to keep a separate copy of it.
- Storage cost of retention, which for vectors is the dominant number and scales with the product of corpus size and the number of retained states.
- Immutability guarantees. Whether a retained state can be altered after the fact, which determines whether it is evidence or merely a copy.
- Deletion reach into retained states, because a versioning scheme that erasure cannot penetrate is a compliance liability rather than an asset.
- Identifier propagation. Whether a state identifier can be carried through retrieval into the answer record without custom plumbing at every hop.
- Configuration capture. Whether the segmentation rule, encoder identity and index parameters are recorded as data rather than living in a deployment that has since been replaced.
- Expiry mechanics. Whether retained states have owned lifetimes, or accumulate until someone notices the bill.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Source-system revision history (document management and wiki version history) | Reconstructing what a document said, without duplicating storage | Retention is the source's policy, not yours; access to old revisions may be restricted differently |
| Versioned table formats (Apache Iceberg, Delta Lake, LakeFS) over extracted text and units | Time-travel queries over the content layer, with snapshot identifiers that are cheap to record | Covers the content, not the index; the vectors for a past state still have to come from somewhere |
| Immutable index generations behind an alias | Serving a specific past index exactly as it was | Storage scales with retained generations; old generations keep old schemas and encoders alive |
| Bitemporal databases (Datomic, XTDB) for the structured layer | Querying as of a past time with both valid and transaction time | Applies to facts and attributes rather than to vectors; a partial answer to the problem |
| Content-addressed artifact storage for extracted text and units | Cheap explanation — what a cited passage said — without retaining whole index generations | Explains but does not reproduce; a past query cannot be re-run against it |
| Configuration and pipeline versioning (version-controlled configuration, DVC, MLflow run records) | Recording which rules and models produced a state, as data rather than as deployment history | Only as good as its coverage; an unrecorded library version is an unexplained difference |
| Answer-side records that store the retrieved passages verbatim | The cheapest route to explaining a specific past response | Duplicates content outside the corpus, including content that later has to be erased |