Why LLMs Should Not Be Your OCR: A Practical Lesson in Document AI

AI Generated Image

For more than 9 years, I’ve been working on digitizing data from images, PDFs using computer vision, OCR engines, and domain-specific extraction pipelines. Over time, we’ve built platforms that can extract data from complex documents and drive fully customized workflows across industries like healthcare, banking, and manufacturing.

Recently, with the rise of multimodal GenAI, I started seeing a new pattern: People began sending entire PDFs directly to LLMs and asking questions about their content. Sometimes, they even added a “Judge LLM” to generate a confidence score.

On paper, this sounds elegant. In practice, it’s risky.

Why This Is Dangerous in Real Systems

These documents often contain critical numerical values.

A single digit error, for example, reading 395 as 899 — is just a “small hallucination” for a model. But in healthcare, finance, or compliance workflows, this can become catastrophic.

Yes, a Judge LLM might tell you the answer is “90% confident” but:

• It’s expensive
• It’s slow
• And most importantly: it’s still judging another probabilistic system

That’s not how you build deterministic systems.

Where the Real Complexity Comes From

The difficulty is not just language. Real-world documents contain:

• Scanned pages
• Tables full of numbers
• Nested structures
• Charts and images
• Mixed layouts

But the biggest problem is scanned content, where the most important information is literally inside pixels.

Experiment: Using LLMs as OCR

My first attempt was straightforward:

1. Convert each page to a high-DPI PNG.
2. Send each image to the LLM.
3. Ask it to behave like an OCR system and return the content faithfully.

The results were very revealing:

✅ Normal text was usually okay

❌ Numbers in tables were frequently wrong

❌ Nested tables were even worse

No matter how I tuned the prompts, the problem remained:

When numbers matter, “almost correct” is simply wrong.

The Deeper Problem: API vs Chat UI Illusion

Another subtle but critical discovery:

LLM APIs do NOT apply the same preprocessing, cleanup, and enhancement steps that consumer tools like ChatGPT UI do.

What looks “magical” in the UI does not automatically happen in production APIs.

This leads many teams to overestimate what they can safely build.

The Architecture That Actually Works

The solution turned out to be boring and extremely robust:

Step 1: Classical OCR (e.g., Google Vision, Azure OCR, etc.)

• Detect layout and structure (tables, blocks, forms)
• Extract text and numbers with high numerical accuracy

Step 2: Generate a faithful intermediate representation

• Convert OCR output into structured Markdown / JSON
• Preserve layout and semantics

Step 3: Then use LLMs but only for:

• Restructuring
• Summarization
• Reasoning
• Formatting
• Explanation

In short: OCR for extraction. LLMs for intelligence.

A Production-Grade Bonus: Confidence Scoring

Now you can:

• Compare OCR output vs LLM interpretation
• Cross-validate critical fields
• Generate fast, cheap, deterministic confidence scores

This is:

• Faster than Judge LLMs
• Cost effective
• And far more consistent

The Bigger Problem in Today’s AI Discourse

We are living in a massive GenAI hype cycle because of which, classical, deterministic techniques are being forgotten. This is a potential threat, in fact an alarming danger.

GenAI is:

• Powerful
• Transformational
• Absolutely necessary

But it is not a replacement for engineering discipline.

Final Thoughts: How Real Systems Should Be Built

GenAI does not replace classical systems. It sits on top of them.

If your problem demands:

• Correctness
• Faithfulness
• Auditability
• Determinism

Then your architecture should look like:

Image Processing + OCR + Validation + LLM + Post-processing

Not:

“Throw PDFs at an LLM and hope for the best.”

Discarding 30 years of document processing science because of GenAI hype is not innovation; it is irresponsible engineering.

The real winners will be those who combine classical methods with modern AI, not those who blindly replace one with the other.

DocumentAI #IntelligentDocumentProcessing #OCR #GenAI #LLM #ComputerVision #AIArchitecture #Quadratyx #Scanalityx #PowerCA