Chunking vs Tokenization: The Two Splits Every AI Engineer Keeps Confusing

Why conflating these two ideas quietly breaks your RAG pipeline — and how to stop doing it

I spent an entire afternoon debugging a RAG pipeline that kept returning half-sentences as “relevant context.” The embeddings looked fine. The retrieval scores looked fine. The vector store looked fine. The bug was upstream, in a place I hadn’t even thought to check: I had built my chunking logic on top of an assumption about tokenization that was just wrong.

That afternoon is the reason this post exists. If you’ve worked on LLM pipelines for more than a few months, you’ve probably used the words “chunk” and “token” almost interchangeably in conversation — “let’s chunk this into 512 tokens,” “how many tokens per chunk,” “token-based chunking.” The language blurs the two concepts together, and that blurring costs people real debugging time.

So let’s separate them properly. Chunking and tokenization solve different problems, operate at different stages of the pipeline, and fail in different ways. Understanding exactly where one ends and the other begins is one of those unglamorous fundamentals that separates people who can reason about their RAG system from people who are just tuning knobs until the eval score goes up.

The one-sentence distinction

Tokenization converts text into the discrete numerical units a model actually understands and processes — it’s a property of the model.

Chunking splits documents into retrievable, digestible pieces of context — it’s a property of your pipeline and your data.

Tokenization happens inside the model’s input layer, every single time, on every string you send it, whether you like it or not. Chunking is a design decision you make, upstream, about how to structure a corpus before anything ever reaches a model. One is fixed and largely invisible; the other is something you architect, tune, and iterate on for weeks.

They interact constantly — a chunking strategy that ignores tokenization behavior will silently produce bad chunks — but they are not the same operation, and treating them as interchangeable is where a lot of subtle RAG bugs come from.

Tokenization: how a model sees text at all

Every transformer-based LLM — GPT-4o, Claude, Llama, whatever you’re shipping with — doesn’t read “words.” It reads a sequence of integers, each one an index into a fixed vocabulary the model was trained with. Tokenization is the deterministic algorithm that maps raw text to that sequence of integers, and it’s decided once, at training time, baked into the model’s tokenizer artifact. You don’t get to choose it per request.

Byte-Pair Encoding (BPE) and friends

Most modern LLMs use some flavor of subword tokenization — BPE, WordPiece, or SentencePiece are the common families. The core idea in all of them: start with characters or bytes, and iteratively merge the most frequent adjacent pairs into new vocabulary entries, until you hit a target vocabulary size (commonly 32k–128k+ tokens).

The practical consequence engineers actually run into:

• Common English words are often a single token: **the, is, model.**
• Less common words get split into subword pieces: **tokenization might become `token+ization`**.
• Rare strings — code identifiers, non-English scripts, emoji, made-up product names — can explode into many tokens per “word,” sometimes even per character.

This is why a sentence in Hindi or Tamil can cost 2–3x the tokens of the semantically equivalent English sentence, and why a variable name like **xgboost_lstm_ensemble_v2** can quietly eat 8 tokens where you budgeted for 2. If you're building anything India-market-facing — say, an LLM layer summarizing NSE/BSE filings or Zerodha order flow logs — this isn't a footnote, it's a cost and context-budget line item.

Why this matters beyond “counting tokens for billing”

Token count isn’t just a pricing detail. It directly determines:

1. Context window consumption. Your 128k context window is 128k tokens, not words, not characters. A “1000-word chunk” is not a fixed cost — its token cost depends on vocabulary, language, and content type (code vs. prose vs. numbers).
2. Attention behavior. Transformers compute attention over token positions. Where you place token boundaries affects how coherently the model can attend across a passage — split a meaningful unit mid-token-boundary in a weird way (rare, but it happens with certain encodings) and you can degrade downstream comprehension.
3. Truncation logic. If your input exceeds the context window, truncation happens at the token level, not the sentence or chunk level, unless you explicitly control for it. Naive truncation can lop off a token mid-word if you’re not careful with the tokenizer’s own boundaries.

The tool every engineer should actually open once

If you’ve never opened OpenAI’s **tiktoken** or Anthropic's tokenizer and actually looked at how your specific production strings get split, do it once. It stops being an abstraction the moment you see your own prompt template getting chopped into pieces you didn't expect — especially around markdown syntax, JSON braces, or code fences, all of which tokenize less efficiently than plain prose.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("pip-compile requirements.in --output-file requirements.txt")
print(len(tokens), tokens)
# CLI flags and file extensions often split into 2-3 tokens each —
# this is exactly the kind of string that silently inflates your token budget

Chunking: how you decide what the model gets to see

Now zoom out from the model to the pipeline. Chunking is the process of breaking a large document — a PDF, a Confluence page, a codebase, a 40-page SEBI circular — into smaller pieces that get embedded, indexed, and retrieved independently. It’s the retrieval layer’s unit of currency.

Chunking exists because of two hard constraints that have nothing to do with tokenization directly:

1. Embeddings degrade with length. Cram too much heterogeneous content into a single vector and it becomes a semantic average of everything in it — good at matching nothing precisely. A chunk needs to represent one coherent idea, not a whole document’s worth of ideas.
2. Context windows are still finite and expensive. Even with 128k+ windows now common, stuffing your entire knowledge base into every prompt isn’t retrieval — it’s just re-implementing full-context inference badly, at higher latency and cost, with worse precision than proper retrieval.

The common chunking strategies, and where they actually fail

Fixed-size chunking. Split every N tokens (or characters), often with some overlap. Dead simple to implement, and it’s where most people start:

def fixed_chunk(text, chunk_size=512, overlap=50):
    tokens = enc.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = start + chunk_size
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))
        start += chunk_size - overlap
    return chunks

The failure mode: it doesn’t know what a sentence is. It will happily cut a chunk boundary in the middle of a clause, splitting a claim from the number that supports it, or a code block from the explanation that makes it make sense. This is precisely the bug I opened this post with — half-sentence chunks that embed into a semantic no-man’s-land.

Recursive character/semantic-boundary chunking. Split preferentially at paragraph breaks, then sentence breaks, then word breaks, only falling back to a hard cut when nothing else fits. LangChain’s **RecursiveCharacterTextSplitter** and LlamaIndex's node parsers work this way. Much better default behavior for prose-heavy corpora — it respects the document's own structure before it respects your size budget.

Semantic chunking. Instead of a fixed size, embed sentences (or small sentence groups) and split at points where cosine similarity between consecutive sentence embeddings drops — i.e., where the topic actually shifts. This produces chunks that are coherent by meaning, not by character count, at the cost of an extra embedding pass over the whole document up front.

Structure-aware chunking. For anything with inherent structure — Markdown headers, code files, API specs, schema docs — chunk along that structure instead of ignoring it. A function definition is a natural chunk boundary in code. An **## H2** section is a natural boundary in a Markdown-based knowledge base. If you're documenting a multi-agent pipeline's README, chunking by section header preserves far more retrievable meaning than chunking by token count ever will.

Agentic / LLM-based chunking. Have an LLM read the document and propose chunk boundaries directly, optimizing for “does this chunk stand alone and answer a plausible question.” Expensive, slow, but genuinely useful for high-value corpora — regulatory filings, contracts, anything where retrieval precision has a real cost of being wrong.

The part where the two concepts collide

Here’s the actual interaction point, and it’s the reason this post exists rather than being two separate, boring glossary entries:

Your chunk size is usually defined in tokens, but your chunk boundaries should be defined by semantics — and those two objectives pull in different directions.

If you set **chunk_size=512** and just cut at the 512-token mark, you're optimizing for a token budget while being blind to meaning. If you split only at paragraph breaks with no size ceiling, you're optimizing for meaning while being blind to the fact that one paragraph might tokenize to 40 tokens and the next to 4,000, blowing your embedding model's max sequence length or making your retrieved context wildly uneven in information density.

Practically, this means good chunking logic does both, in this order:

1. Establish semantic boundaries first (paragraphs, sections, sentences, code blocks).
2. Merge small adjacent semantic units up to a token budget.
3. Only hard-split a single semantic unit if it alone exceeds the token budget — and if you must, tokenize it yourself so you control exactly where the cut lands, rather than trusting a character-count heuristic that doesn’t know what a token boundary even is.

def chunk_by_tokens_with_semantic_bias(sections, max_tokens=512):
    chunks, current, current_len = [], [], 0
    for section in sections:  # pre-split by paragraph/header
        section_len = len(enc.encode(section))
        if current_len + section_len > max_tokens and current:
            chunks.append("\n".join(current))
            current, current_len = [], 0
        if section_len > max_tokens:
            # single section too big — hard split on token boundaries, not chars
            section_tokens = enc.encode(section)
            for i in range(0, len(section_tokens), max_tokens):
                chunks.append(enc.decode(section_tokens[i:i+max_tokens]))
        else:
            current.append(section)
            current_len += section_len
    if current:
        chunks.append("\n".join(current))
    return chunks

That’s the actual fix I made after that debugging afternoon: measure chunk size in the target embedding model’s tokenizer, not an approximate character count, and never let a hard split fall anywhere except a token boundary you control.

Where the two actually sit in a pipeline

The diagram below is the whole argument of this post compressed into one picture. Chunking happens upstream, in your ingestion pipeline, before anything touches a model. Tokenization happens downstream, inside the model’s input layer, on every request, whether you think about it or not.

Chunking (teal) is a decision you make in your pipeline. Tokenization (blue) is something the model does to whatever you hand it.

Two failure modes that look identical but aren’t

If you’re triaging a bad retrieval result, it helps to know which layer to blame:

• “The retrieved chunk is truncated or cut off mid-thought.” That’s a chunking problem. Your boundary logic split a semantic unit. Fix it upstream in the splitter, not by fiddling with the embedding model.
• “The retrieved chunk looks complete, but the model’s response ignores half of it, or the prompt gets truncated unexpectedly.” That’s a tokenization/context-budget problem. Your token accounting is off somewhere — probably you estimated tokens with len(text) / 4 instead of the actual tokenizer, and the real count came in higher than budgeted.

I’ve seen both misdiagnosed as “the embedding model is bad” more times than I’d like to admit. It rarely is. It’s almost always the boundary logic or the token accounting.

A production checklist

If you’re building or auditing a RAG or agentic pipeline, this is the short list I actually go through now:

• Tokenize with the real tokenizer of the model you’re actually calling, not an approximation. tiktoken for OpenAI, the model-specific tokenizer for open-weight models — never len(text) // 4 as your production token counter.
• Measure chunk size in tokens of your embedding model, since embedding models often have a different tokenizer and a harder max-sequence-length ceiling than your generation model.
• Never let overlap and chunk size fight the embedding model’s max input length — overlap eats into your effective budget; account for it explicitly instead of discovering it in a truncation warning log.
• Chunk boundaries should respect document structure first, token budget second. Structure-aware or recursive splitting as the default, fixed-size splitting only for genuinely unstructured text blobs.
• Log both the token count and the chunk boundary decision during ingestion. When retrieval quality drops, you want to be able to look at a chunk and immediately tell whether it was cut for size or cut for structure.
• Re-tokenize when you change embedding models. A chunking strategy tuned against one tokenizer’s token boundaries doesn’t transfer cleanly to another — vocabularies differ, and “512 tokens” means a different amount of actual text depending on whose vocabulary you’re using.

The takeaway

Tokenization is the model asking “how do I even read this string.” Chunking is you deciding “what’s the smallest self-contained unit of meaning worth retrieving on its own.” One is fixed by the model you chose. The other is a design surface you control completely — and it’s usually where the actual engineering leverage in a RAG system lives.

Get tokenization wrong and you get weird truncation bugs and blown budgets. Get chunking wrong and you get a retrieval system that’s technically working and semantically useless — which is a much harder failure to spot, because nothing errors out. It just quietly retrieves the wrong half of the right sentence.

If you’re building production RAG or multi-agent pipelines and want to go deeper on dependency management, tokenizer internals, or chunking strategies for structured data schemas or regulatory filings, drop a comment — happy to write a follow-up.

📢 Follow for More

If you found this helpful, don’t forget to follow **@neuraldev on Medium **💡

Github Link: [https://github.com/UdaySharmaGitHub](https://github.com/UdaySharmaGitHub)

I regularly share content on: Machine Learning • AI • Deep Learning • Agentic AI • LLMs • DSA • Tech Careers 🚀

👉 Like • Share • Comment — and let’s grow together!