Data Connectors — Patterns

High-water mark with a reconciliation sweep

Problem. Incremental crawling by modification timestamp is cheap and is wrong in both directions: sources fail to bump the timestamp for permission changes and some metadata edits, and items deleted outside the window are never revisited.

Approach. Run the incremental crawl continuously against a persisted high-water mark, and on a slower cadence run a full enumeration that compares the set of identifiers the source reports against the set the corpus holds. The sweep emits deletions for identifiers that have disappeared and re-emits items whose permission fingerprint has changed, without re-downloading content whose hash is unchanged.

Trade-off. The sweep costs a full listing pass against a quota you share with real users, so its cadence is bounded by the source rather than by how fresh you want to be. On very large sources a full sweep may be a weekly job, which puts a hard floor under how quickly a missed deletion can be caught.

Permission fingerprint as a first-class change signal

Problem. Content and entitlement change independently, and most sources only report the former. An index that re-reads a document only when its bytes change will serve a revoked document to a departed contractor for as long as the document stays untouched.

Approach. Have the connector compute a stable fingerprint over the resolved entitlement — the sorted set of principals, the inherited container identifiers, and any deny rules — and emit a change event when that fingerprint differs from the stored one, independently of content. Downstream stages can then update the entitlement payload without re-parsing or re-embedding.

Trade-off. Resolving entitlement is itself an API call per item on many sources, so this turns a cheap listing pass into an expensive one. Where group membership rather than the document's grant is what changed, the fingerprint will not move at all, and the gap has to be closed at query time instead.

Source-shaped throttling with a shared budget

Problem. A backfill and the steady-state crawl draw on the same tenant quota, and so do the customer's employees. Treating the connector's throughput as an independent tuning knob produces an outage in a system you do not own.

Approach. Give each source a single token budget that both the backfill and the incremental path draw from, with the backfill assigned the lower priority and allowed to consume only headroom. Honour the source's own retry hints rather than a fixed backoff, and expose the remaining budget as a metric so that a stalled backfill reads as throttling rather than as a failure.

Trade-off. Backfills become unpredictable in duration and can stall indefinitely when interactive usage rises, which makes the initial launch date of a corpus hard to commit to. It also means a genuinely urgent re-crawl has no fast path unless the priority can be inverted deliberately.

Content-addressed external identity

Problem. Identities derived from paths or URLs change when a human reorganises a folder, and the corpus acquires a duplicate of every moved file while the original is never deleted.

Approach. Prefer the source's own immutable item identifier as the primary key and treat the path as mutable metadata. Where a source offers no stable identifier, derive one from a hash of the content plus the smallest stable attribute available, and carry the previous identity in the emitted event so downstream stages can perform a rename rather than an insert and a delete.

Trade-off. Content-derived identities make every edit look like a new document, which is the opposite failure, so the two schemes have to be combined per source rather than chosen globally. Sources that recycle identifiers after deletion will silently attach new content to an old document's history.