RAG and Knowledge Engineering
Data Connectors — Stack
Data connectors › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
You are buying coverage of other people's APIs and a credible story about change and permission capture. Judge on the parts that are expensive to build yourself.
- Change-detection fidelity per source. Does the implementation use the source's own delta or change-notification mechanism where one exists, or does it fall back to timestamp polling everywhere? This single property sets the achievable lag.
- Permission capture. Whether entitlement is extracted at all, whether group membership is resolved or left as opaque identifiers, and whether permission-only changes produce an event.
- Deletion and move semantics. Whether removals arrive as events, and what the component does about items that simply stop appearing.
- Credential model. Service account versus delegated per-user authorisation, token rotation, and whether secrets stay in your boundary.
- Quota awareness. Built-in rate limiting, adaptive backoff, and the ability to cap throughput so a backfill does not degrade the source for its human users.
- Resumability. Whether a crawl interrupted at hour nine resumes at hour nine, and whether cursor expiry is handled rather than silently swallowed.
- Extensibility. How much work a source nobody has written a connector for actually is, and whether the framework's abstractions survive an API that does not fit them.
- Operational visibility. Per-source counts of items seen, emitted, skipped and failed, which is the only way an absence becomes detectable.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Purpose-built RAG connector frameworks (LlamaHub readers, LangChain document loaders, Unstructured connectors) | Breadth of sources and fast prototypes | Coverage is wide and shallow; many readers do full enumeration only and extract no permissions |
| Managed enterprise search and ingestion services (Glean, Elastic connectors, Azure AI Search indexers, Vertex AI Search data stores) | Organisations that need permission-aware crawling of common SaaS without building it | Entitlement model is the vendor's; verify how it handles groups, deny rules and permission-only changes |
| Change-data-capture over the source's own database (Debezium, native logical replication) | Internal systems you control, where the database is reachable | Bypasses application-level permissions entirely; you are now coupled to someone else's schema |
| Event-driven object-store triggers (bucket notification to queue) | File-drop and data-lake corpora | Object stores carry no document-level entitlement; permissions must come from somewhere else |
| Workflow and integration platforms (Airbyte, Fivetran, n8n) with a custom sink | Reusing an existing integration estate and its credential management | Tuned for analytics replication; document-shaped fidelity and ACL capture are usually afterthoughts |
| Direct SDK integration written in-house per source | A small number of sources that carry most of the value | Each one is a standing maintenance commitment against an API you do not control |
| Protocol-mediated source access through a tool gateway | Reusing connections an agent platform already holds | Interactive tool access is shaped for single lookups, not for enumerating millions of items |