From PDFs to Passages — The Art and Science of Chunking

Splitting documents sounds trivial. It is not. Here’s how to do it right so your retrieval model has a fighting chance.

*Series: Fine-Tuning Embedding Models for Domain-Specific Retrieval Part 2 of 4*

Photo by Brett Jordan on Unsplash

Picture this: you’ve got a 40-page academic paper. You feed the entire thing into your embedding model, get back a single vector, and store it in your vector database.

Now a user searches for “what was the experimental setup on the WMT’14 dataset?” That answer lives in one paragraph on page 12. But your vector represents the entire paper — the abstract, the related work, the conclusion, the references, and that one critical paragraph all averaged into a single point in space.

The signal drowns in noise. Your retriever returns the paper, sure, but only because the paper as a whole is somewhat relevant. It can’t tell you which page.

This is why we chunk. Splitting a document into smaller pieces — each embedded as its own vector — restores the signal-to-noise ratio. But how big should the chunks be? Where should you cut? How do you handle PDFs that look like alphabet soup when you naively extract text?

This part is about getting that right. Get chunking wrong and no amount of fine-tuning will save you — a model trained on bad chunks just learns to retrieve bad chunks well.

1. The Three Properties of a Good Chunk

A good chunk is:

1. Small enough that a single vector can represent it faithfully.
2. Large enough to contain a complete, answerable unit of information.
3. Consistently sized across the corpus, so retrieval scores are comparable.

That third one trips people up. If chunk A is 100 tokens and chunk B is 800 tokens, their embeddings live at different “scales” — the longer chunk has more content competing for that single vector. Even after L2 normalization, longer chunks behave differently in retrieval. Consistency matters.

2. The Naïve Approach: Character or Word Splits

The simplest possible strategy: split every N characters or N words.

def naive_split(text: str, chunk_size: int, overlap: int) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += (chunk_size - overlap)
    return chunks

Problems with this:

• Splits mid-sentence, mid-formula, mid-code block. Your model has to embed "the gradient of the loss func" as a complete thought.
• Different documents have wildly different character-to-token ratios. Code is dense (many chars per token), prose is sparse. A 1000-character chunk could be 150 tokens of code or 600 tokens of English.
• The embedding model counts tokens, not characters. Your “chunk size” is meaningless from the model’s perspective.

Character or word splits are never the right choice for an embedding pipeline. Skip them entirely.

3. Token-Aware Sliding Window: The Production Standard

The correct unit is the token, counted by the same tokenizer the embedding model uses.

💡 Why must the tokenizers match? Different tokenizers segment text completely differently. The word “tokenization” is 1 token to a BPE tokenizer trained on technical text but 4 tokens to a tokenizer with a smaller vocabulary. If you chunk with one tokenizer and embed with another, your “384 tokens” might actually be 500 to the embedder — silently truncated, with information lost.

The algorithm has two steps.

Step A: Tokenize the full document, tracking page numbers

token_ids: list[int] = []
token_pages: list[int] = []
for page_idx, page_text in pages:
    ids = tokenizer.encode(page_text, add_special_tokens=False)
    token_ids.extend(ids)
    token_pages.extend([page_idx] * len(ids))
    # Insert a paragraph boundary between pages
    sep_ids = tokenizer.encode("\n\n", add_special_tokens=False)
    token_ids.extend(sep_ids)
    token_pages.extend([page_idx] * len(sep_ids))

By tracking which page each token belongs to, you get precise page-span metadata for every chunk — invaluable for citations (“see page 12”) and debugging.

Step B: Slide a window over the token sequence

target_tokens = 384      # chunk size
overlap_tokens = 64      # overlap between consecutive chunks
min_tokens = 50          # drop trailing stubs smaller than this
stride = max(1, target_tokens - overlap_tokens)  # 320
start = 0
chunks = []
while start < len(token_ids):
    end = min(start + target_tokens, len(token_ids))
    chunk_ids = token_ids[start:end]
    if len(chunk_ids) < min_tokens and start != 0:
        break  # drop trailing stub
    text = tokenizer.decode(chunk_ids, skip_special_tokens=True).strip()
    chunks.append(text)
    start += stride

Why 384 tokens?

Most modern embedding models were trained on passages of 128–512 tokens. Going much larger compresses too much meaning into one vector; going much smaller loses surrounding context. 384 leaves headroom for special tokens and instruction prefixes when you encode up to a 512-token budget.

Why 64-token overlap?

Sentences near a chunk boundary deserve representation in both adjacent chunks. Without overlap, a sentence that straddles a boundary gets its first half in chunk N and its second half in chunk N+1 — neither chunk fully captures the idea.

Document tokens:  [0 .... 383 | 320 .... 703 | 640 .... 1023 | ...]
                              ↑ 64-token overlap ↑

A rule of thumb: overlap = 10–20% of chunk size.

4. The PDF Extraction Problem

PDFs are not text files. They store text as positioned glyphs, not as flowing paragraphs. The PDF spec doesn’t even guarantee reading order. That’s a problem.

Two-column academic papers are the canonical nightmare:

Column 1 (x: 50-290):          Column 2 (x: 310-550):
┌──────────────────────┐       ┌──────────────────────┐
│ We propose a method  │       │ experiments show     │
│ for training dense   │       │ that our approach    │
│ retrieval models...  │       │ achieves SOTA...     │
└──────────────────────┘       └──────────────────────┘

A naïve PDF parser reading top-to-bottom, left-to-right gives you:

“We propose a method experiments show for training dense that our approach retrieval models… achieves SOTA…”

Interleaved gibberish. Your embedding model will dutifully encode it as if it were real text, and your retrieval will silently degrade.

The fix is a PDF library that sorts text blocks by bounding box before reading. With PyMuPDF:

import fitz   # PyMuPDF
doc = fitz.open("paper.pdf")
for page in doc:
    text = page.get_text("text", sort=True)
    # `sort=True` orders text blocks top-to-bottom within each detected column,
    # then left column before right column.

It’s not perfect for every layout, but it handles the majority of academic papers correctly.

Cleaning up extraction artifacts

PDFs love sprinkling in invisible characters and weird line breaks. A simple normalization pass goes a long way:

import re
def normalize(text: str) -> str:
    text = text.replace("\u00ad", "")              # soft hyphen (invisible!)
    text = re.sub(r"-\n(?=\w)", "", text)          # de-hyphenate: "posi-\ntion" → "position"
    text = re.sub(r"[ \t]+", " ", text)            # collapse whitespace
    text = re.sub(r"\n{3,}", "\n\n", text)         # cap consecutive newlines
    return text.strip()

⚠️ The soft hyphen trap. Soft hyphens (Unicode U+00AD) are invisible characters PDFs sprinkle in as line-break hints. They render as nothing, but your tokenizer sees them. The word “posi­tion” with a soft hyphen tokenizes differently than “position”. You can have two chunks of “the same” text producing different embeddings purely because of invisible characters. Strip them.

5. Deduplication: The Silent Killer of Contrastive Training

Many real-world PDF corpora are riddled with repeats: boilerplate copyright notices, page headers/footers on every page, identical preprint and published versions of the same paper, slide decks recycled across courses.

Duplicates are catastrophic for contrastive training. Why?

If chunk A and chunk A’ are near-identical and end up in the same training batch:

query → chunk A     ← labeled "positive"
query → chunk A'    ← labeled "hard negative" (because it came from a different doc)

You’re now asking the model to simultaneously pull A close to the query and push A’ far away — even though A and A’ say the same thing. The training signal contradicts itself, the loss never converges cleanly, and your model gets confused.

The fix is dead simple: hash the normalized text and skip duplicates.

import hashlib, re
def hash_text(text: str) -> str:
    norm = re.sub(r"\s+", " ", text).strip().lower()
    return hashlib.md5(norm.encode("utf-8")).hexdigest()
seen = set()
unique_chunks = []
for chunk in all_chunks:
    h = hash_text(chunk)
    if h in seen:
        continue
    seen.add(h)
    unique_chunks.append(chunk)

For larger corpora, layer MinHash LSH on top to catch near-duplicates (paraphrased boilerplate, version differences).

6. Sentence-Aware Chunking

Token-window chunking can sometimes split mid-sentence — which is fine 95% of the time, but ugly. Sentence-aware chunking fixes this by snapping boundaries to sentence endpoints.

import spacy
nlp = spacy.load("en_core_web_sm")
def sentence_chunks(text: str, target_tokens: int, tokenizer) -> list[str]:
    sentences = [s.text.strip() for s in nlp(text).sents]
    chunks, current_tokens, current_text = [], [], []
    for sent in sentences:
        sent_tokens = tokenizer.encode(sent, add_special_tokens=False)
        if len(current_tokens) + len(sent_tokens) > target_tokens and current_text:
            chunks.append(" ".join(current_text))
            current_tokens, current_text = sent_tokens, [sent]
        else:
            current_tokens.extend(sent_tokens)
            current_text.append(sent)
    if current_text:
        chunks.append(" ".join(current_text))
    return chunks

This produces chunks that always start and end at sentence boundaries. The trade-off: chunk sizes become more variable, and very long sentences (e.g., a citation-heavy academic sentence) can push you over your budget.

For most use cases, token-window chunking is fine. Sentence-aware chunking is worth it when your downstream consumer (a reranker or LLM) is sensitive to fragments.

7. Semantic Chunking: Splitting Where Topics Change

A more sophisticated approach: embed each sentence, then split wherever consecutive sentences are semantically dissimilar. The idea is that a topic change is the natural place to chunk.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
def semantic_chunks(sentences, embedder, threshold=0.5):
    embeddings = embedder.encode(sentences)
    breaks = [0]
    for i in range(1, len(embeddings)):
        sim = cosine_similarity(embeddings[i-1:i], embeddings[i:i+1])[0, 0]
        if sim < threshold:
            breaks.append(i)
    breaks.append(len(sentences))
    return [sentences[start:end] for start, end in zip(breaks[:-1], breaks[1:])]

Trade-offs:

• Chunks respect natural topic boundaries.
• 2× embedding cost (you embed during chunking and again for retrieval).
• Sensitive to the similarity threshold.
• Wildly variable chunk sizes.

Semantic chunking shines for documents with clearly delineated sections on different topics — legal contracts, product manuals, multi-topic FAQs. It’s overkill for academic papers, where section headers already provide structure.

8. Chunking Code

Source code has very different structural properties from prose. You don’t want to split a function in half. Use the language’s AST instead:

import ast
def code_chunks(source: str) -> list[str]:
    tree = ast.parse(source)
    chunks = []
    lines = source.split("\n")
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            chunks.append("\n".join(lines[node.lineno - 1 : node.end_lineno]))
    return chunks

For multi-language codebases, tree-sitter handles Python, JavaScript, Go, Rust, and dozens of others with the same API.

9. Picking the Right Chunk Size: An Empirical Process

There is no universally correct chunk size. It depends on:

1. Query length distribution. Short keyword queries match well against smaller, focused chunks. Long natural-language questions match better against larger, richer passages.
2. Document structure. Dense academic text packs more meaning per token than conversational Slack chat. 384 tokens of arXiv ≠ 384 tokens of Slack.
3. The embedding model’s training distribution. Models trained on 128–512-token passages will be slightly off-distribution at 1024 tokens, even if their context window allows it.

A practical tuning recipe:

# 1. Take a representative sample of your corpus.
# 2. Embed it at several chunk sizes.
# 3. Evaluate retrieval quality (Recall@5) against your existing query set.
# 4. Pick the smallest size that doesn't sacrifice recall.
#    (Smaller = less compression = more faithful embeddings.)
results = {}
for chunk_size in [128, 256, 384, 512, 768]:
    chunks = chunk_corpus(docs, chunk_size=chunk_size)
    recall = evaluate_retrieval(chunks, queries)
    results[chunk_size] = recall
# Plot, pick the elbow.

For most ML/scientific PDF corpora, 384 tokens with 64-token overlap is an excellent starting point.

10. The Split That Everyone Gets Wrong

This is a correctness issue most tutorials get backwards.

Wrong: randomly split your chunks 80/10/10 into train/val/test.

Why it’s wrong: chunks from the same document end up in both train and test. The model has effectively seen the document’s content during training, then is “evaluated” on near-duplicates. Your validation metrics are wildly optimistic, and you’ll be heartbroken when you deploy.

Correct: split by doc_id, not by chunk.

import numpy as np
def make_splits(chunks_df, ratios={"train": 0.8, "val": 0.1, "test": 0.1}, seed=42):
    doc_ids = chunks_df["doc_id"].unique()
    rng = np.random.default_rng(seed)
    rng.shuffle(doc_ids)
    n = len(doc_ids)
    train_end = int(n * ratios["train"])
    val_end = train_end + int(n * ratios["val"])
    return {
        "train": set(doc_ids[:train_end]),
        "val":   set(doc_ids[train_end:val_end]),
        "test":  set(doc_ids[val_end:]),
    }

At evaluation time you still rank queries against the full chunk corpus (including training documents) — because at deployment time, the corpus contains everything. Only the queries come from held-out documents; the model has never seen the ground-truth passages those queries point to.

11. The Output Schema

After chunking, you want a tidy table that looks like this:

Field Type Purpose chunk_id string unique identifier, e.g. "paper42::0023" doc_id string source document, for split-by-doc doc_path string absolute path to original file page_start int for citation page_end int for citation text string the actual chunk content n_tokens int tokenizer count, for training sanity checks

Having n_tokens in the table lets you sanity-check before training:

import pandas as pd
chunks = pd.read_parquet("chunks.parquet")
print(chunks["n_tokens"].describe())
# Most chunks should hit your target size.
# Small ones at the bottom are end-of-document stubs.

If you see chunks much longer than your target, something went wrong with your tokenizer or your decode step.

12. Putting It All Together

Here’s the chunking pipeline as a single flow:

PDF files
   │
   ▼
[Extract text — sort by bounding box]
   │
   ▼
[Normalize — strip soft hyphens, de-hyphenate, collapse whitespace]
   │
   ▼
[Tokenize whole document, tracking page numbers]
   │
   ▼
[Slide 384-token window, stride 320]
   │
   ▼
[Decode tokens back to text]
   │
   ▼
[Hash + deduplicate]
   │
   ▼
chunks table → ready for embedding

It’s a lot more than text.split("\n\n"), but every step earns its place.

🔑 Key Takeaways

• Chunk in tokens, not characters. Use the same tokenizer as your embedding model.
• Overlap matters. 10–20% of chunk size keeps boundary sentences whole.
• PDFs are evil. Sort by bounding box, de-hyphenate, strip soft hyphens.
• Deduplicate early. Duplicates create contradictory training signals that silently kill contrastive learning.
• Split by document, not by chunk. Otherwise your offline metrics will lie to you.
• There’s no universal chunk size. For ML/scientific text, 384 tokens with 64 overlap is a solid default — but always validate empirically.

Next up — Part 3: Recall, MRR, NDCG, and the Metrics That Actually Tell You If Your Retriever Works.