A Multimodal Model Can Read a PDF and Miss the Document

Direct PDF input makes document automation look like a one-call problem. Upload a file, request JSON, and let a multimodal model inspect both words and pixels. That is newly practical for contracts, invoices, research papers, and compliance packets that defeated plain text extraction because their meaning depended on layout.

The catch appears after the first successful demo. A model may accurately transcribe every visible value on one page while assigning a table cell to the wrong column, detaching a footnote from its marker, or treating a repeated header as fresh content. Page reading and document parsing are different jobs.

The eight-page packet that exposed the gap

For a controlled test, I built an eight-page synthetic purchasing packet with known answers. Pages 2 and 3 contained one table split across the page break. Page 4 was a scan rotated 90 degrees. Page 5 used superscript footnote markers, while page 6 instructed the reader to “see Table 2 on the following page.

” The same company header appeared in the same position on all eight pages.

I ran the packet through three routes. The first extracted the PDF text layer, the invisible character data embedded behind many digital pages, while retaining each word’s page number and coordinates. The second rendered every page at 300 dots per inch and sent the images to a vision-capable model. The third combined both representations and asked for the same JSON schema.

This was a failure-mode test, not a benchmark. The fixture was designed to answer a narrower implementation question: which representation should control each part of the output?

The text-only route preserved exact strings from the digital pages and made the repeated header easy to detect by position. It struggled where reading order mattered. Some table values arrived as plausible text in the wrong sequence, and the rotated scan contributed nothing because it had no usable text layer.

The image route handled the rotated page and understood the table’s visible columns better. It also treated pages as largely self-contained surfaces. The split table could become two tables, the repeated company header could enter the extracted body eight times, and the “following page” reference remained unresolved unless the prompt and supplied context explicitly linked pages 6 and 7.

The hybrid route had enough evidence to fix those errors, but only after the application performed the document work around the model. The model did not infer a dependable hierarchy merely because it could see the page.

Page images win on local visual meaning

A rendered page is the stronger input when meaning depends on marks that text extraction cannot represent. The 90-degree scan is the clearest case. Optical character recognition, or OCR, converts pixels into machine-readable text, but it must first detect the page orientation; a vision model can often read the rotated image directly, while a dedicated OCR step can rotate it deterministically before extraction.

Tables create a more qualified win. A page image preserves the fact that “$125” sits below “Unit price” and beside a particular line item, even when a basic text extractor emits all product names first and all prices later. That makes image input useful for recovering column associations and merged cells.

Yet the image should not automatically control the final value. Small decimal points, superscript markers, and similar-looking characters can be easier to preserve from a clean PDF text layer than from pixels. In the purchasing packet, the useful division of labor was visual interpretation for the cell relationship and text extraction for the literal string.

Footnotes need the same split. The image reveals that a superscript “2” is attached to a particular sentence and that the corresponding note sits below a rule at the bottom of page 5. Coordinate-aware text extraction supplies the exact note text and its bounding box, which is a rectangle describing where the text appeared. A parser can then connect marker and note using page position before asking the model to interpret the note’s effect.

Image calls carry a cost beyond model pricing. The application must render pages, upload larger inputs, and process more visual tokens than a short text representation would require. Sending all eight pages as images also consumes context that could have held neighboring clauses or validation instructions. For born-digital pages with a clean text layer, that expense buys little.

Text and coordinates win on document structure

Repeated headers show why visual comprehension is insufficient. A model looking at one page has no statistical basis for knowing that a company name at the top is boilerplate. The application does: text appearing at nearly the same coordinates across most of an eight-page file is a header candidate and can be labeled before model inference.

Cross-page references are even less visual. The phrase “following page” describes a relationship between pages 6 and 7, while “see Table 2” points to a named object that may be several pages away. Neither relationship exists inside the pixels of page 6. The parser must maintain page order, identify table labels, and build links between references and targets.

The split table on pages 2 and 3 required another document-level rule. Page 3 repeated the column headings but did not repeat the table title. A page-image model could reasonably produce a second object. The hybrid parser instead compared the columns, observed the adjacent page numbers, and joined the rows under the table identifier established on page 2.

This distinction matters for retrieval-augmented generation, where an application fetches selected document passages before asking a model to answer. If ingestion stores page 3 as an independent table, a later question about the full order may retrieve only half the rows. The extraction looked readable, but the downstream answer is incomplete.

Build a parser that chooses per page

Start with a preflight pass rather than a model call. For every page, record whether a text layer exists, how many characters it contains, the page rotation, image coverage, and word coordinates. A page with dense embedded text should follow a different route from page 4 of the test packet, which was effectively one large rotated image.

Next, extract the native text with coordinates and preserve the PDF’s page boundaries. Do not flatten the entire file into one string. Label recurring text near the top or bottom as probable headers and footers, but retain it as evidence instead of deleting it immediately; a repeated line can still contain a changing page number or section label.

Render only pages that need visual interpretation. Useful triggers include missing text, a large scanned region, low OCR confidence, or clusters of words whose coordinates suggest a table. A 300-dpi render was enough for the controlled packet, though small print may require a higher setting and therefore larger uploads.

Send the model a page image together with the extracted words, their coordinates, and a constrained output schema. For a table, request a table identifier, column names, row values, footnote markers, and continuation status. The schema narrows the task, while the supplied text lets the model copy exact strings rather than retranscribe every character from pixels.

Then reconcile. If the image interpretation places a value under “Unit price” but the text layer supplies a different character sequence at the same coordinates, retain both candidates and apply field-specific validation. A date can be checked for a valid calendar form; a subtotal can be compared with its rows. Validation should flag disagreement rather than asking the same model to certify its own answer.

After page extraction, run a document pass that sees the ordered page summaries, heading hierarchy, table identifiers, repeated regions, and unresolved references. This is where the parser joins the table from pages 2 and 3, links page 6 to Table 2 on page 7, and suppresses the eight header instances from body retrieval.

Finally, store evidence with every extracted field. At minimum, keep the source page, bounding box, extraction route, and original text or image crop. That record makes human review practical: an operator can open the exact cell or footnote rather than searching the full PDF after a validator reports a conflict.

The adoption decision

Use text extraction alone when files are born digital, reading order is uncomplicated, and the application mainly needs search or summarization. It is cheaper to run, easier to audit, and less likely to alter exact strings.

Use page images when scans, handwriting, rotation, diagrams, or complex tables carry the meaning. Even then, keep page numbering and neighboring context outside the model call. Pixels restore local layout; they do not create a document graph.

The hybrid route earns its extra engineering cost when wrong relationships matter more than occasional transcription errors, particularly in document-heavy workflows that feed databases, approvals, or retrieval systems. The eight-page packet needed image interpretation on the rotated scan and table pages, while its headers and cross-page links were better handled by coordinates and deterministic rules. Rendering all eight pages would have cost more without fixing the structural problem.

Questions people ask

Can a multimodal model understand a PDF table?

It can often infer columns, merged cells, and visual grouping from a page image better than plain text extraction. For dependable output, pair that interpretation with native PDF text or OCR, retain coordinates, and validate values before joining tables that continue onto another page.

Should every PDF page be converted to an image?

No. Born-digital pages usually provide cleaner strings through their embedded text layer, with lower processing and upload costs. Render pages selectively when preflight detects scans, rotation, missing text, or layout whose meaning cannot be recovered from coordinates alone.

Why do repeated headers appear in AI summaries?

A page-level model sees the header as visible text each time and may not know it repeats across the file. Detect text at matching coordinates across multiple pages, label it as a header, and exclude it from body retrieval while preserving the original evidence.

How should cross-page references be handled?

Keep pages in order, identify named objects such as tables and sections, and resolve references in a second document-level pass. A page image can read “see Table 2,” but the surrounding application must locate Table 2 and attach the link to the extracted claim.