Chunking for RAG: Why How You Split Your Documents Determines Everything

180 chunks became 104. Retrieval quality improved immediately. Here’s the decision that made it happen — and the framework to get it right.

Most RAG tutorials spend 80% of their time on the language model and 20% on retrieval. The retrieval section usually covers embeddings, vector databases, and similarity search. Chunking gets a paragraph — maybe a code snippet setting **chunk_size=512** — and moves on.

This is backwards.

Chunking is the decision that determines what your model is allowed to see at query time. Get it wrong and no retrieval technique, no reranker, no prompt engineering fix will fully compensate. The damage is structural.

This article covers what chunking is, why it matters more than most engineers realize, the four strategies from prototype to production, and a decision framework built from actual build experience — including going from 180 raw chunks down to 104 clean ones, with measurable retrieval improvement before changing anything else.

— -

## What chunking actually is

A language model can only process a limited amount of text at once. When you build a RAG system, you store your documents as a collection of smaller pieces — called chunks — each embedded as a vector in a database.

At query time, the user’s question gets embedded, and the system retrieves the most similar chunks. Those chunks are passed to the language model as context, which then generates an answer.

The chunk is what the model sees. Not the document. The chunk.

Every retrieval decision flows from this: the quality of your chunks sets a ceiling on the quality of every answer your system can produce. A model can only answer well if the retrieved chunk contains the right information, in the right scope, with the right context.

Chunking is not a preprocessing step. It is the decision that determines what the model is allowed to see — and what it will never find.

— -

## The newspaper analogy

Before getting into mechanics, here’s a mental model that makes chunk size intuitive.

Imagine cutting a newspaper to answer a question. You have three approaches:

Too small: You cut every sentence into its own slip of paper. Someone asks, “Why did the market crash?” You hand them the slip that says “the market fell.” It’s technically true. It’s completely useless. No context survived the cut.

Too large: You hand them the entire newspaper. The answer is in there somewhere. So is the weather forecast, the sports scores, the opinion section, and the crossword. The model drowns in irrelevant content. Signal-to-noise collapses.

Just right: You cut at paragraph breaks. Each slip contains one complete idea. You hand them the right slip and they have everything they need — nothing missing, nothing extra.

The ideal chunk is a complete, self-contained unit of meaning. One idea. Enough context for that idea to be understood independently.

— -

## What chunk size looks like in practice

Chunk size is measured in tokens (roughly 0.75 words each). Here’s how different sizes behave in practice:

Notice that semantic chunking has no fixed token count — it produces variable-length chunks based on where meaning actually shifts, not where an arbitrary counter expires.

— -

## The four chunking strategies

### 1. Fixed-size splitting (Simplest)

Split on every N characters or tokens, regardless of meaning. Often combined with overlap — a configurable number of tokens shared between adjacent chunks.

chunk_size = 512
overlap = 50

Where it works: Prototypes, quick experiments, homogeneous text with low structure (transcripts, plain prose with no sections).

Where it breaks: Cuts mid-sentence constantly. Overlap is a workaround that duplicates content, inflates your index, and introduces retrieval noise. The underlying problem — arbitrary boundaries — doesn’t get solved.

### 2. Recursive character splitting (Better)

Tries to split on paragraphs first. If a paragraph is too long, splits on sentences. If a sentence is still too long, splits on words. Only goes smaller when forced.

This is the default strategy in most RAG frameworks because it respects natural document structure better than fixed-size splitting while remaining fast and cheap.

Where it works: General-purpose baseline when you don’t know your document type yet. Mixed documents.

Where it breaks: Still produces arbitrary boundaries when paragraph or sentence lengths are uneven. Doesn’t understand meaning — only structure.

### 3. Semantic chunking (Best quality)

This is where chunking becomes a retrieval decision rather than a text operation.

The approach:

1. Embed every sentence individually
2. Compute cosine similarity between adjacent sentences
3. When similarity drops sharply — when a meaning shift is detected — mark a chunk boundary
4. Group sentences between boundaries into chunks

The result: variable-length chunks where each chunk contains one coherent idea, because the boundaries are placed at actual transitions in meaning rather than at arbitrary token counts.

The real-world result: Starting from 180 fixed-size chunks, semantic chunking produced 104 clean chunks. 76 chunks eliminated — sentence fragments, orphaned clauses, mid-thought splits that carried no usable information. Retrieval eval scores improved immediately, before any change to retrieval technique.

Where it works: Any document where retrieval quality matters more than indexing speed. The right choice for production when your query load is high and retrieval errors are costly.

Where it breaks: Slower at index time (you’re embedding every sentence, not just the final chunks). More expensive computationally. Not worth the overhead for prototype work.

### 4. Document-aware splitting (Production)

Use the document’s own structure as chunk boundaries: headers, sections, tables, code blocks, markdown ## markers, page breaks, form fields.

For structured documents, this is almost always the highest-quality strategy, because the author of the document has already made the structural decisions for you. A section header exists precisely because the topic changed. Respecting that boundary is free information.

For PDFs: layout parsing (tools like AWS Textract or pdfminer) extracts heading structure.

For Markdown: split on heading levels.

For code: split on function or class definitions.

Where it works: Any well-structured document — technical documentation, contracts, research papers, product manuals.

Where it breaks: Unstructured text (transcripts, emails, free-form notes) has no structure to exploit. Falls back to recursive splitting.

— -

## The decision framework

Before choosing a chunking strategy, answer these four questions in order:

1. Is your document structured?

Does it have headers, sections, markdown headings, or code blocks? If yes → document-aware splitting. Never cross a structural boundary. The document structure is your best chunking signal.

If no → continue.

2. What kind of questions will users ask?

Focused, specific questions (“What is the refund policy?”) → smaller chunks, 128–256 tokens. You need precision; a small chunk containing exactly the right fact retrieves better than a large chunk where that fact is buried.

Broad, summary-seeking questions (“What are the key themes in this document?”) → larger chunks, 512+ tokens. You need coverage; small chunks lose the broader context needed to synthesize an answer.

3. Is retrieval quality worth the indexing cost?

If retrieval errors are expensive (customer-facing products, legal or medical applications, high-stakes automation), semantic chunking pays for itself. Index once; query thousands of times.

If you’re in prototype or experimentation mode, recursive splitting with 10–15% overlap is a reasonable fast baseline.

4. Do you have an evaluation set?

If yes: A/B test chunk sizes and strategies. Measure recall@k — what percentage of relevant chunks appear in the top k results for your benchmark queries. The right chunk size is empirical, not theoretical.

If no: Build the eval set before optimizing chunking. Chunking without measurement is guessing. A well-constructed eval set of 50–100 representative query-document pairs is worth more than any amount of intuition.

— -

## What tutorials never tell you

Overlap is a patch, not a solution.

Adding overlap to fixed-size chunks is the standard recommendation in most tutorials. It duplicates content across adjacent chunks, inflates your vector store, and introduces retrieval noise — the same content might retrieve twice, competing with distinct relevant chunks. Overlap exists only to paper over mid-sentence cuts. The real fix is smarter boundaries.

Chunk size and embedding model are tightly coupled.

all-MiniLM-L6-v2 has a 256-token input limit. If your chunks are 512 tokens, the end of every chunk gets silently truncated when you embed it. You lose half your content — invisibly, with no error or warning. Always match your chunk size to your embedding model’s actual token limit before you build your index.

The right chunk size is document-specific.

Research papers chunk differently than customer support transcripts. Legal contracts chunk differently than product documentation. Engineering READMEs chunk differently than medical notes. There is no universal right answer. The correct chunk size for your system is determined by your documents and your queries — not by default parameters in a framework.

Fewer clean chunks beats more noisy chunks.

104 semantic chunks outperformed 180 fixed-size chunks. This runs counter to the instinct that more data is better. It isn’t. Every noisy chunk — every sentence fragment, orphaned clause, or mid-thought split — is a retrieval failure waiting to happen. It will occasionally score high similarity to an unrelated query and pollute the context window with useless content. Precision of chunk boundaries matters more than raw chunk count.

— -

## The numbers that matter

Going into semantic chunking, the starting state was 180 fixed-size chunks from recursive splitting. After semantic chunking:

• 104 chunks survived
• 76 chunks eliminated (42% of the index was noise)
• Every surviving chunk contained one complete, self-contained idea
• Retrieval eval scores improved before any downstream change

The improvement came entirely from better boundaries. Same documents. Same embedding model. Same vector store. Same retrieval logic. Only the chunk boundaries changed.

That’s the leverage point.

— -

## Where this fits in the AI engineering stack

Embeddings (Concept 01) are the primitive. Chunking (Concept 02) is the first place you use that primitive in a way that permanently shapes your system’s recall ceiling.

The sequence is:

1. Choose your embedding model (sets recall ceiling)
2. Choose your chunking strategy (determines what gets embedded)
3. Build your vector index
4. Implement retrieval
5. Measure and iterate

Steps 1 and 2 are made once and are expensive to change. Every document needs to be re-chunked and re-embedded if either decision changes. This is why getting them right early — with an eval set, with measurement, with a decision framework rather than defaults — is the highest-leverage investment in any RAG build.

— -

## Summary

Chunking is the decision that determines what your retrieval system is allowed to find. It is not a preprocessing step. It is a core architectural choice with permanent consequences.

• Fixed-size: Fast, cheap, breaks at boundaries. For prototypes only.
• Recursive: Better structure awareness. Good general baseline.
• Semantic: Variable-length, meaning-based boundaries. Best retrieval quality.
• Document-aware: Use the document’s own structure. Best for structured content.

The right strategy depends on your documents, your queries, your quality requirements, and your indexing budget. Measure with an eval set. Don’t guess.

104 clean chunks beat 180 noisy ones. Precision wins.

— -

📂 Source code: https://github.com/Raghuveer030706/ai-engineering-journey