Stop Chunking Your Documents Before You Embed Them

Your RAG retrieval is probably broken, and the default chunking recipe everyone copies is the reason why.

Your RAG system keeps returning chunks that are almost right but missing the one detail that mattered. You blame the embedding model. You swap vector databases. You tune the top-k. None of it moves the needle.

The problem is upstream of all of that. It is how you split the document.

For years the recipe was copy-paste identical everywhere. Take a document, cut it into 512-token pieces, embed each piece on its own, store the vectors. That recipe quietly destroys meaning, and almost nobody notices until retrieval quality plateaus.

The bug that isn’t in your code

Here is the failure. You chunk a paragraph that says “Dr. Smith joined in 2019. She led the migration.”

The second chunk becomes “She led the migration.” Embedded alone, that vector has no idea who “she” is. The pronoun points at nothing. The context lived in the previous chunk, and you threw it away the moment you split first and embedded second.

This is called context fragmentation, and it is everywhere in production systems. Every pronoun, every “this approach,” every “the above method” becomes a floating reference with no anchor. Your embeddings are technically fine. They are just describing text that lost its meaning during preprocessing.

What late chunking actually does

Late chunking flips the order of two steps, and that is the whole trick.

Instead of splitting the document and then embedding each fragment, you embed the entire document first using a long-context model. Because of self-attention, the token vectors for “She led the migration” already carry the “Dr. Smith” signal from earlier in the text. Only after the model has seen everything do you split the token embeddings into chunks and pool them.

The chunks come out the same size. Storage cost is identical. But each vector now remembers the context it came from. Retrieval scores jump because the meaning survived.

The difference between a confused RAG pipeline and a working one is the order of two steps.

The code that shows the difference

Here is the pattern using a long-context embedding model. Encode the full text once, then pool over chunk spans afterward.

from transformers import AutoModel, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("jinaai/jina-embeddings-v3")
model = AutoModel.from_pretrained("jinaai/jina-embeddings-v3", trust_remote_code=True)
def late_chunk(text, spans):
    # spans = list of (start_token, end_token) for each chunk
    inputs = tok(text, return_tensors="pt", truncation=True, max_length=8192)
    with torch.no_grad():
        token_embeddings = model(**inputs).last_hidden_state[0]
    chunk_vectors = []
    for start, end in spans:
        # pool the token vectors AFTER full-document attention
        chunk_vectors.append(token_embeddings[start:end].mean(dim=0))
    return torch.stack(chunk_vectors)

The naive version would tokenize each chunk separately and embed it in isolation. This version lets every token see the whole document before you cut anything. That single reorder is the entire improvement.

The night I chased the wrong bug

I spent a full weekend convinced our legal-doc retriever had a bad embedding model. Contract clauses kept coming back with the right topic but the wrong party attached. I benchmarked three embedding models, rebuilt the index twice, and even started drafting a message to switch vector databases.

Then I actually printed the chunks. Half of them started with “Such party shall” and “The foregoing provision,” pronouns and back-references pointing at clauses that lived in the previous chunk. The model was fine. My chunking had guillotined every sentence away from its subject. I rewrote the pipeline to late chunking on Monday morning and retrieval accuracy went up more than any model swap ever gave me. I felt equal parts relieved and stupid.

When it is worth it, and when it isn’t

Late chunking shines on dense, reference-heavy text. Legal, medical, financial, technical docs where sentences constantly point backward. That is exactly where naive chunking hurts most.

If your documents are short and self-contained, like product FAQs or standalone snippets, plain recursive chunking at 400 to 512 tokens is still fine. Do not add complexity you do not need.

But if your retrieval feels almost right and you cannot figure out why, print your chunks before you touch anything else. The answer is usually sitting right there at the start of chunk number two.

Reorder two steps in your pipeline and you fix a class of bugs no model upgrade will touch. What is the retrieval bug you spent days chasing before realizing it was your data all along?