OCR and Multimodal Extraction

What it is

Recovering retrievable meaning from material that contains no machine-readable text at all: scanned pages and photographs of documents, screenshots pasted into tickets, charts and diagrams whose message is entirely in their geometry, product images, and recorded audio or video where a transcript is the only thing an index can hold. Two different operations live here — transcription, which claims to reproduce what is written, and description, which produces a new account of what something shows — and they have very different standing as evidence.

Why it is not document parsing

Parsing recovers a structure that is already present in the file and can in principle be verified against it. This capability manufactures text that did not exist in the artifact, from pixels or audio, with a confidence that is never one. That changes the obligations: output needs a confidence signal, needs to be labelled as derived, and needs a defined relationship to the original region so that a reader can be shown the thing the text came from. A parser's mistakes are structural; this stage's mistakes are fabrications that read exactly like facts.

Why it matters

In most enterprise corpora the material that is hardest to read is the material people most want answers about — signed contracts held only as scans, engineering drawings, the screenshot of the error that the ticket is actually about, the slide whose entire content is a diagram. Skipping it produces a corpus that is quietly biased toward whatever happened to be typed. Including it badly is worse: a generated description of a chart that invents a figure becomes an indexed sentence, and the next stage downstream has no way to tell it apart from a quotation.

Key decisions

• When this stage runs at all: on every page, only where no usable text layer exists, or gated by a confidence estimate. Running everywhere is the largest per-page cost in most pipelines; running only on empty pages misses files that carry a bad text layer from a previous pass.
• Transcription or interpretation for non-textual content. A transcription can be quoted and cited; an interpretation is a model's claim about an image and must be marked as one wherever it is later retrieved.
• Whether per-token confidence is retained, and what low confidence causes — indexing anyway, indexing with a flag, routing to review, or dropping. Discarding confidence makes every downstream quality question unanswerable.
• How a figure is represented in the index: a generated caption, extracted axis labels and data values, the surrounding prose that references it, the image itself for a later model to read, or several of these.
• Script and language detection, and whether the engine selected per document is recorded, because a wrong-script transcription produces output that is confidently wrong rather than empty.
• The economics of the backfill. Per-page pricing that is tolerable for new documents may make reprocessing a historical archive infeasible, which turns an upgrade into a permanent two-tier corpus.
• What happens with redaction and sensitive imagery, since a transcription step can recover text that a visual redaction was meant to remove, and a description step can state what a face or a badge shows.

Failure modes

• A document carrying a poor text layer from an earlier transcription is classified as digital-native, never re-read, and contributes garbled text that still embeds to something plausible.
• A model describing a chart states a value that is not in the chart, the sentence is indexed, and it is later retrieved and cited as though it came from the document.
• Skewed or rotated scans produce long runs of nonsense that pass every length check, and the only visible symptom is that queries about those documents return nothing useful.
• Tabular content in a scan is transcribed left to right across rules, so every row becomes a single line and the column association is gone before any later stage sees it.
• Handwritten annotations — the signature, the initialled amendment, the pencil note that changes the meaning — are either dropped or hallucinated into confident text.
• Generated descriptions are stored without any marker distinguishing them from transcribed text, so a citation resolves to a sentence that appears nowhere in the source document.
• The per-page cost of the chosen engine makes a full reprocess unaffordable, the backlog is deferred, and the corpus permanently contains two qualities of extraction with no record of which document got which.
• Transcription recovers text beneath a black redaction rectangle that was drawn over, rather than removed from, the original.

Related capabilities

• Document Parsing — handles the files and regions that already contain text
• Citation and Grounding — must distinguish transcribed from generated text
• Chunking Strategies — receives output with no sentence structure to rely on
• Ingestion Pipeline — where the cost and retry policy for this stage is enforced
• Knowledge Freshness — a deferred backlog here is a freshness problem elsewhere

Applies: Production readiness · Design questions