Why Most RAG Systems Fail on Code and How AST-Based Chunking Fixes It

A deep dive into why naive code chunking breaks retrieval and how Abstract Syntax Trees enable smarter, structure-aware RAG systems

The Problem Shows Up Only on Real Code

Most RAG systems feel impressive until you try them on real codebases. You ask a simple question and the answer comes back incomplete or wrong. The issue is not the model it’s how we feed code into the system.

Why the Usual RAG Pipeline Breaks

A typical pipeline looks something like this:

Code → Chunk → Embed → Retrieve → Generate

This works great for documents. Text is naturally sequential, so even rough chunking works fine.

But code isn’t like that.

Code has structure. Things depend on each other. Meaning is tied to blocks, not just lines.

And when we ignore that, things start breaking in subtle ways.

Where Things Actually Go Wrong

Most systems chunk code using token limits or line splitting. This creates broken pieces of functions.

When embedded, the model learns incomplete context, leading to noisy retrieval and unreliable answers.

The Core Mistake: Treating Code Like Text

The real issue is simple.

We are treating code like text.

But code isn’t text. It’s structured.

That structure is what gives code its meaning, and once we lose it, everything downstream suffers.

Understanding AST in a Practical Way

This is where AST (Abstract Syntax Tree) comes in.

Instead of thinking in terms of lines, AST represents code as a tree of logical components functions, classes, loops, and expressions.

So now, instead of random chunks, we actually know what a complete unit of meaning looks like.

The Shift: Chunk by Structure, Not Size

The idea that follows is pretty straightforward.

Instead of chunking by size, we chunk by structure.

Functions stay intact. Classes stay intact. Logical boundaries are preserved.

This makes each chunk self-contained and meaningful, which is exactly what a model needs.

How This Works in Practice

In practice, this is simpler than it sounds.

You start by parsing code into an AST using something like *Tree-sitter.*

Then you walk through the tree and identify logical units like functions and classes.

If a unit is too large, you break it into smaller nodes. If it’s too small, you merge it with nearby ones.

So you’re constantly balancing between size and meaning.

What Changes After This

Once you do this, things start improving in a very noticeable way.

Retrieval becomes more accurate because chunks actually represent real logic.

The model gets full context instead of fragments.

And the answers feel more reliable not because the model changed, but because the input finally makes sense.

Why This Matters More Than You Think

This directly affects everything built on top of code.

RAG systems become more trustworthy.

Agents can reason better because they see complete functions instead of partial snippets.

Even simple code search starts feeling more intelligent.

The Insight Most People Miss Even I Had

Most people optimize models, embeddings, and prompts. But chunking defines what the model actually sees.

Final Thought

If your chunks are broken, your system is broken.

AST-based chunking shifts from treating code as text to structured logic — and that changes everything.

If you’re building RAG systems, agents, or developer tools, this is worth getting right early.

References & Credits

• Research Paper: https://arxiv.org/abs/2506.15655v2
• Blog: https://supermemory.ai/blog/building-code-chunk-ast-aware-code-chunking
• Tree-sitter Docs: https://tree-sitter.github.io/tree-sitter/

Images used in this blog are sourced from Google for illustrative purposes.

If you want to explore further or try it yourself: https://github.com/supermemoryai/code-chunk