RAG and Knowledge Engineering

RAG and Knowledge Engineering

Data Connectors

2 of 81 in this book · 3 min read

What it is

The per-source component that authenticates to an external system of record, enumerates what that system holds, and emits each item together with the two things the retrieval corpus cannot reconstruct later: who was allowed to see it, and when it last changed. Every connector speaks one vendor's dialect — its pagination, its quota, its idea of a modification timestamp, its export formats — and its whole job is to translate that dialect into a uniform stream of documents, permission facts and change events.

Why it matters

The ceiling on what a retrieval system can ever answer is set here. A folder a connector cannot enumerate does not exist as far as every downstream stage is concerned, and the absence shows up as a confident, incomplete answer rather than an error. Permissions have the same property in reverse: the source system is the only place where entitlement is authoritative, so a connector that drops or flattens it hands every downstream filter a guess to enforce. Connectors are also where an AI product acquires a long tail of operational obligations to systems it does not own — quotas it can exhaust, tokens it must rotate, and APIs that deprecate on someone else's schedule.

Key decisions

  • How change is learned: polling a listing endpoint, subscribing to the source's change feed, or consuming a delta cursor where the vendor offers one. A cursor is authoritative and often has an expiry; a modification timestamp is convenient and frequently not updated for the changes that matter most, such as a permission edit.
  • Which identity the connector uses. A broad service account enumerates everything and therefore needs permissions captured separately and correctly; per-user delegated credentials make entitlement implicit but multiply the crawl by the number of users and break when someone leaves.
  • What fidelity to take. Many sources offer the original bytes, a rendered HTML view, and a plain-text field, and they differ in structure, completeness and licence. Choosing the cheap field early is a decision that is expensive to revisit across a crawled corpus.
  • Whether moves, renames and soft deletes are modelled as events or inferred from absence. Inference requires a full enumeration to be safe, which many large sources make impractical.
  • How the backfill is throttled relative to the source's human users, given that the quota is usually shared with the people whose work the connector is reading.
  • What the stable external identity of an item is, and whether it survives a rename. Deriving it from a URL or a path guarantees duplicates the first time someone reorganises a folder.

Failure modes

  • The connector runs as an administrator so that it can enumerate everything, and the private drafts it was never meant to reach are indexed alongside the published material.
  • A document's permissions are tightened but its content is untouched, the source does not bump the modification timestamp, and the stale entitlement stays in the corpus indefinitely.
  • A pagination cursor expires partway through a multi-day backfill, the crawl restarts from the beginning, and the portion between the restart point and the previous high-water mark is never revisited.
  • Sustained crawling exhausts a tenant-wide API quota, and the first symptom is that the customer's own employees cannot use the SaaS product the connector is reading.
  • Items in a trash or recycle state are still returned by the listing API and are indexed as live, so deleted material keeps answering questions.
  • A folder reorganisation changes every path, the connector mints new identities for unchanged files, and the corpus doubles with no stage reporting an error.
  • An OAuth refresh token expires during a holiday, ingestion stops, and because the pipeline only alarms on failures rather than on an absence of change events, nobody notices for weeks.

Related capabilities

Applies: Production readiness · Design questions