Chunking in RAG

So I was working on a project in the F&B industry where I had to take menu input and ingest it into my database. After that, whenever the user makes a query, it retrieves the relevant information from the database and gives it to the LLM, which then generates the answer for the user in a more defined way.

A simple flow of the system looked like this:

User Query → Retriever → Vector DB → Relevant Dish Data → LLM → Response

While working on this, one thing I learned deeply was that chunking strategy depends a lot on the type of data you are working with, especially whether the data is structured or unstructured.

The Problem With Traditional Chunking

Initially, I was not aware of the JSON method by which we can chunk information row-wise. Till then, I mostly knew traditional chunking methods used for PDFs and unstructured documents, where chunking is usually done by splitting text into paragraphs, sentences, or character-based chunks using methods like recursive splitting.

But here the problem started.

Suppose there is a dish in the menu:

• Coffee
• Medium sweet
• $5

Now if I apply normal PDF-style chunking, it may split the information like this:

• [Coffee, medium sweet]
• [$5]

In this case, the relation and context of the dish start getting lost. Because price, sweetness level, or attributes of the dish are correlated with that specific dish, splitting them can affect retrieval quality and sometimes even increase hallucinations.

Yes, chunk overlap can help to some extent, but it still does not completely solve the problem because the dish relationship itself may still break.

Why I Switched to JSON-Based Chunking

So from here, I switched to structured JSON-based ingestion.

Instead of chunking randomly, I handled the data dish-wise (row-wise). Each dish became a structured unit containing all related information together.

Example:

json { “dish”: “Coffee”, “sweetness”: “Medium sweet”, “price”: “$5” }

This gave me much better control over the data.

Since every dish had metadata attached to it, I could also do metadata filtering during retrieval.

For example:

• filter by category
• filter by price range
• filter by dish type

Now, technically metadata filtering is possible even with PDFs, but it becomes much harder unless additional preprocessing and structuring are done. With JSON, it becomes much more manageable and organized.

Because of this, retrieval quality also improved since the complete dish context stayed together instead of getting fragmented into unrelated chunks.

What I Learned About Chunking in RAG

One major thing I learned from this project is:

For structured and organized data, traditional chunking is often not the best option.

If the data is naturally relational (like restaurant menus, product catalogs, tabular information, inventory, etc), storing it in structured formats like JSON gives much better control over:

• relationships between fields
• metadata filtering
• retrieval quality
• context preservation

On the other hand, for unstructured data (PDFs, articles, documentation, research papers, long text) normal chunking methods such as RecursiveCharacterTextSplitter work much better because the information is not organized in a strict row-wise relationship.

There, semantic or paragraph-based chunking makes more sense.

Final Thought

This project made me understand that chunking is not just splitting text it is about preserving meaningful relationships in data for better retrieval.

The better the chunking strategy based on the data type, the better the retrieval quality in a RAG system.