Memory and State Management
Memory Provenance — Stack
Memory provenance › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
You are choosing how derivation relationships are represented, queried in both directions, and kept honest over time.
- Reverse traversal. Whether "everything derived from this source" is a cheap query, since that is the question retraction and erasure both ask.
- Chain depth handling. Whether multi-generation derivation is representable without the query cost exploding at depth three.
- Write-path overhead. Provenance is written on every memory write; a representation that doubles write cost will be disabled under load.
- Tamper evidence. Whether a record can be shown not to have been altered, which matters only for some use cases and costs in all of them.
- Schema stability. Whether new origin types and new derivation kinds can be added without a migration, because the taxonomy will grow.
- Survivability of references. What happens to a provenance edge when its target is deleted, and whether the representation distinguishes "deleted" from "never existed".
- Content leakage. Whether the representation stores identifiers or copies of source text, since the latter makes the provenance store an independent retention problem.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Source identifier columns on the memory row | The minimum viable provenance; cheap to write and to read | Single parent only; multi-source derivations and chains cannot be expressed |
| Edge table of derived-from relationships (PostgreSQL with recursive CTEs) | Most systems: multi-source derivation with reverse traversal in ordinary SQL | Recursive queries degrade at depth; the edge table needs its own indexes and its own retention rule |
| Graph store for lineage (Neo4j, FalkorDB, Memgraph) | Deep or heavily branching derivation graphs where traversal is the dominant query | A second datastore to operate and keep consistent with the memory store |
| W3C PROV-compatible records (PROV-O in an RDF store, PROV-JSON documents) | Organisations that need an interchangeable, standards-based lineage representation | Verbose and modelling-heavy; most of the vocabulary will go unused for this purpose |
| Data-lineage platforms (OpenLineage, Marquez, DataHub, OpenMetadata) | Teams that already run lineage for pipelines and want memory derivation in the same view | Built around datasets and jobs, not individual claims; per-item granularity is an awkward fit |
| Append-only log of memory write events (Kafka, an event store, an audit table) | Reconstructing what was written, by what, in what order, after the fact | Reconstruction is a batch exercise; it answers history questions, not live retrieval-time ones |
| Signed or hash-chained records (transparency-log style structures, in-toto-style attestations) | Cases where provenance must be defensible against tampering by insiders | Key management and verification overhead; rarely justified for ordinary product memory |