Knowledge Freshness — Patterns

Canary documents measuring the real path

Problem. Every internal proxy for freshness — job success, processing duration, queue depth — stays healthy through the failures that matter most, because a connector that returns nothing looks exactly like a source where nothing happened.

Approach. Write a small, uniquely identifiable document into each connected source on a fixed schedule, carrying the time it was written. Query for it periodically through the normal retrieval path and record the interval between its source timestamp and the moment it first becomes retrievable. That interval is the only number that reflects the whole chain, and its absence past a threshold is an alarm on the source rather than on a stage.

Trade-off. Canaries require write access to production systems that teams are reluctant to grant, and they must be excluded from user-facing retrieval without being excluded from the measurement path, which is a narrow distinction that is easy to get wrong. They also measure the path a small text document takes, which is the fastest path, so they understate the tail.

Age carried onto the unit and into the answer

Problem. Freshness measured in a dashboard does not help the person reading an answer. At the moment of use, the system has a passage and no notion of how old it is, so it presents a withdrawn policy exactly as it presents a current one.

Approach. Store the source modification time and the indexing time on every unit, return both with every retrieved passage, and give the answer layer a rule: material past its source's target is either labelled with its age in the response, or withheld, depending on the document class. The reader sees "as of March 2024" rather than an undated assertion.

Trade-off. Age labels on everything are noise, and labelling only the stale items means the absence of a label is an implicit freshness claim the system may not be able to honour. Source modification times are also frequently wrong — bulk migrations reset them all — so the label can be confidently incorrect.

Explicit supersession rather than coexistence

Problem. When a new version of a document is ingested without any relationship to the old one, both compete in retrieval, and the old one frequently wins: it has been linked to more, it is often longer, and its terminology matches queries written when it was current.

Approach. Model supersession as an edge between units: the new version records what it replaces, and the superseded unit is marked. Retrieval excludes superseded units by default, admits them only when a query is explicitly historical, and the answer layer is told when it has been given one. Where the source does not express versioning, derive it from a stable document identity plus the effective-date attribute.

Trade-off. Detecting supersession without a source signal is heuristic, and a false supersession silently removes a live document from the corpus. Keeping superseded units also costs storage and keeps content retrievable that some deletion obligations would require to be gone.

Backlog age as the alarm, with per-source liveness

Problem. Queue depth falls when a connector stops producing work, so the standard pipeline alarm improves at the exact moment the corpus starts going stale. Aggregate alarms also hide one dead source among twenty healthy ones.

Approach. Alarm on the age of the oldest unprocessed change per source, and separately on the time since the last successful source poll regardless of whether it found anything. The first catches a pipeline that is falling behind; the second catches a connector that has stopped asking. Both are per source, and both are compared against that source's own stated target rather than a global one.

Trade-off. Per-source thresholds multiply the configuration and go stale as sources are added and their change rates shift, so they need an owner. Sources with genuinely sporadic change also produce alarm fatigue on the liveness signal unless the heartbeat is separated from the change feed entirely.