Document Parsing — Patterns

Route by document profile rather than by file extension

Problem. A single parser applied to everything is either too slow for the eighty percent that is plain prose or too weak for the twenty percent that carries the tables people actually ask about. File extension says almost nothing: a PDF may be digital-native, a clean scan, a scan with a bad pre-existing text layer, or a mix.

Approach. Probe each document cheaply first — character density per page, presence and quality of an embedded text layer, column count, table-like region count — and dispatch to a parser chosen for that profile. Record the profile and the parser identity on the document so the population can be audited later and so a parser change can be scoped to the affected profile.

Trade-off. The probe is another stage to maintain and its thresholds drift as corpus composition changes. Misrouting is invisible by construction: a document sent to the cheap path produces output, just worse output, and only sampling catches it.

Preserve an element tree, flatten late

Problem. Once parsing emits a string, heading hierarchy, table structure and element boundaries are gone, and no downstream stage can recover them. Yet most consumers want text.

Approach. Make the parser's contract a list of typed elements — heading with level, paragraph, table with cells, caption, list item, figure — each carrying its anchor and its parent path. Serialise to text only at the point of use, so segmentation can split on element boundaries, a citation can resolve to an element, and a table can be rendered differently for the encoder than for the reader.

Trade-off. The element schema becomes an interface that every parser must satisfy and every consumer must handle, including element types only one parser produces. It is also larger to store and slower to serialise than a string, which matters at corpus scale.

Table extraction as a separate, verified sub-pipeline

Problem. Tables fail differently from prose: the failure is a correct-looking number attached to the wrong label, and it is invisible to every length-based or language-based quality check.

Approach. Detect table regions, extract them with a tool chosen for tables specifically, and verify structurally before accepting — column counts consistent across rows, numeric columns parseable, totals reconciling where a total row exists. Store the table as structured cells alongside a rendered textual form, and on verification failure retain the region as an image for a later interpretation stage rather than emitting flattened text.

Trade-off. Verification rules are corpus-specific and catch only the tables that happen to be self-checking. Keeping both a structured and a rendered form duplicates storage and creates two representations that can disagree about what the table says.

Boilerplate detection by repetition across the corpus

Problem. Hand-written rules for headers, footers, watermarks and navigation chrome are per-source, go stale, and are the first thing nobody maintains. Left in place, the repeated text is present in a large fraction of fragments and degrades the discrimination of every one of them.

Approach. After parsing, count exact and near-exact line repetition within a document and across documents from the same source, and strip lines whose repetition rate exceeds a threshold and whose position is consistent — top or bottom of page, start or end of a page's element list. Keep what was stripped, attached to the document, so the decision is reversible and auditable.

Trade-off. Legitimately repeated content — a safety warning on every page of a manual, a standard clause in every contract — looks identical to boilerplate by this measure, and removing it deletes exactly the text some queries are about. The threshold is a corpus-level judgement that needs re-checking whenever a large new source is added.