RAG and Knowledge Engineering

RAG and Knowledge Engineering

Knowledge Graph

54 of 81 in this book · 3 min read

What it is

A maintained store of typed entities and typed relations with explicit identity, provenance and a schema — the organisation's structured account of what things exist and how they relate, independent of any particular use made of it. Building one from documents means three things that are usually conflated: deciding the schema, extracting assertions from text, and deciding when two extracted mentions are the same thing.

Why it matters

Documents state facts in prose, redundantly, inconsistently and across time. A graph is where those statements become addressable: a single node for a supplier rather than four hundred mentions, an edge that says this contract governs that service, a record that the relation was asserted by a specific paragraph of a specific document on a specific date. That addressability is what makes a fact correctable, deletable, auditable and joinable with structured systems — none of which a corpus of passages supports. It is also an artifact with a maintenance burden unlike an index: an index can be rebuilt from its sources at any time, and a graph that has accumulated human corrections cannot.

Key decisions

  • Whether the schema is fixed before extraction or allowed to emerge from it. A constrained schema yields a graph that can be queried and validated and discards whatever does not fit; an open one captures everything and produces thousands of near-synonymous relation types.
  • The identity policy. How two mentions become one node, which attributes are decisive, and crucially whether a merge can be undone — because an incorrect merge creates assertions neither original entity ever made.
  • Whether provenance is recorded per edge. Without it the graph cannot be corrected, audited or erased from, and a disputed assertion has no author.
  • How contradictions between sources are represented: resolved to one value, held as competing assertions with their sources, or scoped by the period each was true for.
  • Whether the graph is derived or authoritative. A derived graph must be rebuildable from its sources, which forbids storing human corrections in it unless those corrections are themselves recorded as inputs.
  • How temporal validity is modelled, given that most extracted relations were true when written and many are not now.
  • What quality signal exists. Node and edge counts grow monotonically whether extraction is improving or degrading, so sampling and judging edges is the only real measurement.

Failure modes

  • Edges carry no reference to the text that asserted them, so a request to remove a document leaves its assertions in the graph with no way to find them.
  • Two people with the same name are merged into one node, and the resulting entity has two employers, two nationalities and a career that never happened — and the merge cannot be undone.
  • Open extraction produces "supplies", "is_supplier_of", "provides_to", "vendor_for" and forty more, so every query must enumerate synonyms and every traversal misses some of them.
  • The graph is rebuilt from scratch on each extraction run, and the corrections a domain expert spent a week making are silently discarded.
  • No temporal qualification is stored, so a former chief executive is asserted as the current one, with a citation to a document that was accurate when published.
  • A model-extracted attribute is subtly wrong across a whole document class — an amount parsed in the wrong currency — and the error is uniform, confident and invisible in aggregate statistics.
  • Extraction is re-run with an upgraded model and produces a different graph with different node identities, breaking every stored reference to it.
  • The ontology is designed by a modelling team and the extraction is tuned by an engineering team, and the two never reconcile, so most extracted assertions fail validation and are dropped.

Related capabilities

Applies: Production readiness · Design questions