RAG and Knowledge Engineering
OCR and Multimodal Extraction — Stack
Ocr and multimodal extraction › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
Measure on your own worst pages, not on clean ones, and treat cost per page as a primary criterion rather than a footnote.
- Accuracy on your document population, reported as character and word error rate on a held-out sample that includes your bad scans, your fonts and your languages.
- Confidence output. Whether per-word or per-region confidence is exposed at all, since without it there is no gate between usable output and noise.
- Structure preservation in images. Whether tables, forms and key-value pairs come back with geometry, or as a flat sequence of recognised words.
- Script and language coverage including mixed-script pages, right-to-left text and vertical layouts, weighted by what is actually in the corpus.
- Transcription versus description. Whether the tool claims to reproduce text or to explain an image, and whether that distinction is visible in its output.
- Cost and throughput per page at backfill scale, which often decides the architecture before accuracy does.
- Pre-processing requirements — deskew, denoise, binarisation, resolution floors — and whether they are built in or yours to supply.
- Data residency. Whether scanned contracts and identity documents may be sent to a hosted service at all.
- Determinism and versioning, because a model-based extractor that changes behind a stable endpoint silently changes a corpus that was built from it.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Classical OCR engines (Tesseract, OCRmyPDF as a wrapper) | Local, free, deterministic transcription of clean printed text | Accuracy falls sharply on poor scans; table and form structure is not recovered |
| Modern open-weight OCR stacks (PaddleOCR, docTR, Surya, EasyOCR) | Self-hosted accuracy on harder pages, with layout and script coverage | GPU capacity planning becomes part of ingestion; model revisions change output |
| Hosted document-intelligence APIs (Azure AI Document Intelligence, Google Document AI, AWS Textract) | Forms, key-value extraction and tables in scans at production accuracy | Per-page pricing dominates backfills; documents leave your boundary; schemas evolve |
| Vision-language models (frontier multimodal APIs, open-weight multimodal models) | Charts, diagrams, screenshots and anything needing interpretation rather than transcription | Produces fluent claims that may not be in the image; always label output as generated |
| Speech-to-text systems (Whisper and derivatives, hosted transcription APIs) | Meeting recordings, calls and video where the transcript is the retrievable artifact | Speaker attribution and timestamps are what make a transcript citable; verify both are produced |
| Chart and figure data extraction tools | Corpora where numeric series inside images are the thing being asked about | Narrow coverage of chart types; failure is a plausible wrong series, not an error |
| Image pre-processing libraries (OpenCV, ImageMagick, Leptonica) | Deskew, denoise and resolution normalisation ahead of any engine | Over-aggressive binarisation destroys faint text that the engine would otherwise have read |