RAG for Document AI: Why the Contract Clause You’re Looking For Keeps Getting Lost

Part of the series: Building document AI that holds in production

I’ve seen the same failure play out across different teams enough times to recognize it before they do. An ML team builds a retrieval-augmented generation system over a corpus of contracts. They test it, the recall numbers look reasonable. Then a lawyer asks: which contracts have automatic renewal clauses with less than 30 days notice? The system returns three contracts. The actual answer is 47.

The chunks were too large. The renewal clause was split across a chunk boundary. The retrieval step did exactly what it was designed to do. The design itself was wrong.

This is the RAG failure nobody warns you about when you’re following a general-purpose tutorial. Building RAG on top of documents is different from building RAG on top of news articles, help pages, or Wikipedia. Documents have structure. Contracts have clauses that span pages and reference other sections. Invoices have interdependent fields. Insurance forms have sections where the meaning of one field depends entirely on another. When you apply generic chunking logic to structured documents, you build a system that works on demo queries and breaks on the one that actually matters to the business. Read more about it here.

TL;DR

RAG feeds relevant document chunks to an LLM to ground its answers in real data rather than training memory. But standard RAG chunking strategies ignore document structure, which means key clauses get split, metadata gets lost, and retrieval precision falls below what legal, operations, or compliance teams will accept. Building document RAG that holds requires semantic or structure-aware chunking, metadata-filtered vector stores, and hybrid retrieval combining dense and sparse search. Adaptive chunking in clinical decision support achieved 87% accuracy versus 50% for fixed-size approaches. That gap is not theoretical.

Why documents break general-purpose RAG

Standard RAG assumes text is largely homogeneous. A Wikipedia article, a research paper, a customer support thread. You chunk it, embed it, and retrieve it. The structure is optional context.

Documents violate this assumption in every direction.

A contract’s indemnification clause might read: “Vendor shall indemnify and hold harmless Client from all third-party claims arising from Vendor’s performance of Services, provided that Vendor’s liability under this indemnification shall not exceed the total fees paid in the preceding 12 months.” That’s one clause. Two sentences. A fixed 512-token chunk boundary might split them. A query for “What is Vendor’s maximum liability?” retrieves only the cap without the indemnification scope it limits. The LLM answers confidently and incorrectly.

Metadata is another dimension that breaks generic pipelines. A financial document’s date, counterparty, and currency aren’t just annotations — they’re query filters. If you’re asking about 2024 renewal terms, you don’t want to retrieve clauses from a 2018 contract that happens to use similar language. Generic RAG has no way to enforce this. Every chunk from every contract competes equally in vector space.

At enterprise scale, the problem multiplies. Fifty thousand contracts at 100 chunks each is five million vectors. Even strong embeddings retrieve noise at that size without metadata partitioning. The system slows, precision falls, and the legal team stops trusting it.

The six stages where document RAG can fail

Building document RAG that actually works means getting six stages right in sequence, because failures compound downstream.

Document processing and structuring comes first. Every document must be readable and classified before any chunk is created. OCR quality sets the ceiling for everything downstream — a scanner artifact in stage one propagates through embedding and retrieval. Document classification matters because you cannot chunk an insurance policy the same way you chunk an invoice. A structured document pipeline that identifies document type before chunking is not optional infrastructure; it’s the foundation the whole system depends on.

Chunking strategy is where most pipelines actually fail. Fixed-size chunking is easy to implement and wrong for documents. Semantic chunking splits at logical boundaries — paragraphs, sections, clauses — rather than token counts. Structure-aware chunking uses document metadata like headers and table boundaries to inform where splits happen. Hierarchical chunking creates chunks at multiple granularities, so a clause is a chunk, the full section is also a chunk, and retrieval picks the appropriate level based on query type. Research comparing 36 different chunking approaches across document types consistently shows structure-aware and semantic approaches outperforming fixed-size splits. On clinical documentation, adaptive chunking reached 87% accuracy; fixed-size reached 50%.

Embedding model selection interacts with chunking in ways most teams discover too late. A large general-purpose model like text-embedding-3-large or Vertex AI text-embedding-004 captures rich semantic meaning but costs more at indexing and query time. Domain-specific models like BioBERT or FinBERT add vocabulary precision for highly specialized corpora but introduce operational overhead. The honest starting point for most teams: use a strong general-purpose model, benchmark it on 100 representative queries from your actual corpus, and only switch to a domain model if retrieval precision is demonstrably below threshold.

Vector store design determines how well metadata filtering works at query time. Your vector store needs to hold more than vectors — it needs the extracted fields from each document as queryable metadata. Contract date, counterparty name, document version, source section. A query for “renewal clauses signed after January 2024” should filter on metadata first, then run vector search within that subset. Without this, you’re doing vector search across your entire corpus for every query. Storage costs also scale fast: 50,000 contracts at 768-dimensional embeddings across 100 chunks each runs to roughly 3.8 terabytes. Partitioning by document type and date range keeps query latency and cost manageable.

Retrieval strategy is where dense versus sparse versus hybrid becomes a real decision. Dense (vector) search captures semantic similarity — useful when queries use different vocabulary than the source documents. Sparse search (BM25, TF-IDF) catches exact phrase matches — critical when a lawyer asks for “automatic renewal” verbatim and needs the exact phrase retrieved. Hybrid combines both. For most document corpora with technical, legal, or financial vocabulary, hybrid retrieval outperforms either approach alone. Running a benchmark on 100 queries before committing to a strategy takes an afternoon and prevents months of accuracy problems.

Generation and grounding is the final stage. The LLM needs to be instructed explicitly to cite its source chunks, to answer only from retrieved content, and to return a “not found” result when the chunks don’t support an answer. Without grounding enforcement, the LLM fills gaps with plausible-sounding inference. For documents where a missed liability clause or overlooked exclusion carries real consequences, a confident wrong answer is worse than an honest “I couldn’t find this.”

Where structured extraction feeds into RAG

The cleanest way to reduce hallucination in document RAG is to give the LLM extracted fields rather than raw chunks whenever possible. An extracted invoice total is ground truth. A chunk containing what might be a total field is probabilistic. Read about it in detail here.

When document AI extracts structured fields before RAG ingestion, two things happen: chunking improves because the system knows which structural zones exist and where their boundaries are, and retrieval improves because the extracted fields become metadata filters. A contract management system that extracts signature date, counterparty name, and contract type from every document can answer “show me renewable contracts signed with vendors in the Northeast in 2024” with a metadata filter before a single vector search runs.

This is the integration that makes enterprise document RAG actually precise rather than approximately useful.

What’s the most expensive RAG failure you’ve seen on a document corpus — was it a chunking problem, a retrieval problem, or something that happened in generation? Drop it in the comments.

Earlier in this series: Few-Shot Model Training · What is AI OCR? · Agentic Document Workflows

Tags: Artificial Intelligence · Machine Learning · Document Processing · Automation · Fintech