Data Deletion Propagation

What it is

Removing a source document's content from every artifact a retrieval pipeline derived from it — extracted text, units, vectors in each index and each replica, lexical postings, graph nodes and edges, generated captions and summaries, caches, evaluation fixtures and retained corpus states — and being able to demonstrate afterwards which systems were reached. The difficulty is not the deletion; it is enumerating the derivatives, several of which no longer contain the original words.

Why it is not access filtering

Restricting who may retrieve a document leaves it indexed, encoded and present; removing it means it is not there for anyone. The two are routinely conflated during incident response, because suppressing retrieval is fast and looks like the same outcome from the outside. It is not: the content remains in the vectors, in the lexical index, in any summary that paraphrased it, and in every backup taken since. A filter is the right answer to an entitlement change and the wrong answer to an erasure obligation, and the second is frequently implemented as the first.

Why it matters

Ingestion exists to copy content into new shapes — that is the whole function of the pipeline — and each shape is a place a deletion has to reach. Some of them contain the content in a form no identifier search will find: a vector is a lossy encoding of text that no longer exists anywhere else, a region summary paraphrases documents it never names, a cached answer quotes a passage that has been removed. Discovering this during an audit, after a completion report has been sent, is considerably more expensive than designing for it. And unlike most pipeline defects, this one has a deadline attached and an external party who will ask.

Key decisions

• What triggers removal: a source deletion event, a subject's erasure request, a retention expiry, or a permission change that the organisation has decided to treat as removal. They differ in scope and in what evidence they require.
• Whether every derived artifact records the source it came from. This is the decision that makes the whole thing computable, and it has to be made at build time, long before the first request arrives.
• Whether removal is immediate suppression plus an asynchronous purge, or a single operation. Suppression bounds exposure within seconds; only the purge discharges the obligation.
• What happens to derived text that paraphrases rather than quotes — a graph edge, a community summary, a generated caption. Regenerating from what remains is the only option that preserves utility, and it is a job rather than a delete.
• Whether retained corpus states, evaluation fixtures, backups and replicas are inside the boundary, and what the externally stated commitment says about restores.
• What evidence is produced per request — which systems were searched, what was found, what was removed, when — and how long that evidence is itself kept.
• How completion is determined, given that reporting at enqueue time makes the report routinely false.

Failure modes

• Units are deleted from the vector index and the lexical index keeps its postings, so the document remains fully retrievable through the other arm.
• Vectors are marked deleted and the segment is not compacted, so the content continues to occupy candidate slots and influence results until a rebuild that may be months away.
• A graph edge asserted solely by the deleted document survives because nothing linked the edge to its source, and traversal keeps returning it.
• A generated summary paraphrases the removed content, contains none of its identifiers, and is never located by any search the deletion job performs.
• The live corpus is cleaned and the retained prior states, the evaluation fixture and the staging environment are not, each of which is a complete copy.
• A replica is rebuilt from a snapshot predating the deletion and silently restores everything that was removed.
• The job is asynchronous, completion is reported when it is enqueued, and the statement sent to the requester is wrong by days.
• Removal is implemented as an entitlement change, so the content stays in the index, in the embedding geometry and in every backup, and the organisation believes it was erased.

Related capabilities

• Metadata and ACL Filtering — restricting who may see content that still exists
• Data Connectors — must emit removals rather than let them be inferred from absence
• Knowledge Versioning — retained states are copies this process has to reach
• Knowledge Graph — per-edge provenance is what makes removal executable there
• Memory Deletion — erasing an agent's own recorded memories, a different population and a different derivation graph

Applies: Production readiness · Design questions