Document Parsing

What it is

Turning a file's bytes into text that still carries the structure the document had — reading order, heading hierarchy, table row and column association, list nesting, footnotes, captions — together with anchors that locate each fragment back in the original, such as a page number or a region on that page. The measure of this stage is fidelity to the document's logical structure, which is a much harder target than recovering the characters.

Why it matters

Everything downstream inherits whatever this stage produces and has no way to detect that it is wrong. A table flattened into a run of numbers still embeds, still retrieves, and still gets quoted back to a user with the wrong figure attached to the wrong label. Reading order recovered from a PDF's content stream rather than from its visual layout produces grammatical, plausible sentences assembled from two different columns. These are not parse errors in any sense the pipeline can observe; they are silently incorrect content that will be cited as authoritative. Structure is also the only thing that lets a later stage keep a section intact or attach a heading path to a fragment, so losing it here forecloses options everywhere else.

Key decisions

• Layout-aware analysis versus stream extraction. Layout analysis recovers columns, reading order and table geometry at substantially higher cost per page; stream extraction is fast and is wrong on anything that is not single-column prose.
• How tables are represented downstream: linearised prose, a delimited grid, markup that preserves cells, or retained as an image for a later stage to interpret. Each choice decides what kinds of question about that table can ever be answered.
• Whether a structural tree is preserved and passed on, or whether the output is a flat string. A flat string makes structure-aware segmentation impossible no matter what the next stage does.
• Which anchors are carried for every fragment — page, region, heading path, element index — and whether they survive the rest of the pipeline intact, since a citation can only be as precise as the anchor recorded here.
• What counts as boilerplate: navigation chrome, repeated headers and footers, cookie banners, watermarks, revision stamps. Removing too much deletes content; removing nothing lets the same forty words dominate every fragment in the corpus.
• Whether the parser is deterministic. A model-based parser that produces slightly different text on each run changes every content-derived identifier downstream and turns an incremental update into a full rebuild.
• How non-body content is treated — tracked changes, comments, speaker notes, hidden columns, alternative text. Each is either valuable context or a disclosure of something that was never meant to be published.

Failure modes

• A two-column layout is read as one, so lines from the left and right columns interleave and the resulting sentences are fluent and meaningless.
• Cells are emitted in sequence with no row or column association, and an answer confidently reports the right number under the wrong label.
• A scanned page inside an otherwise digital file yields no characters, the file is marked parsed because the other pages produced text, and the content of that page never enters the corpus.
• Running headers and footers are retained, so a four-hundred-page manual contributes the same company name and document title to every fragment, flattening the differences between them.
• The parser is swapped for a better one and, because fragment identifiers are derived from the text, the entire corpus is invalidated at once with no plan for the re-encoding cost.
• Tracked changes and reviewer comments in a contract are extracted as body text, and deleted clauses are retrieved as though they were in force.
• Footnote markers are glued to the preceding word, producing tokens that match nothing and splitting terms that should have matched.
• An encrypted or corrupt file returns an empty string rather than an error, and a successful parse of nothing is recorded as success.

Related capabilities

• OCR and Multimodal Extraction — what handles the pages this stage cannot read
• Chunking Strategies — the consumer of the structure preserved here
• Citation and Grounding — can only resolve to anchors this stage recorded
• Ingestion Pipeline — owns the retry, quarantine and reprocess behaviour around this stage
• Knowledge Graph — extraction quality depends on the structure recovered here

Applies: Production readiness · Design questions