RAG: The quality of responses starts with the quality of documents

In the Retrieval Augmented Generation (RAG) system, the organization and structuring of documents are crucial for its efficiency. The quality of document ingestion and understanding directly affects the system’s ability to retrieve relevant information and generate accurate, contextualized responses.

Careful preparation of documents, including their hierarchical organization, AI-optimized rewriting and categorization, significantly improves indexing and search performance (retrieval) and the consistency of the responses generated by limiting the tricks to make existing documents more understandable. This approach requires a thorough understanding of segmentation techniques (chunking), document structures, and contextualization strategies that facilitate access to information for language models.

Segmentation techniques

Among the key steps in the pipeline, document segmentation — i.e., the division of content into units that a language model can use — plays a central role. This subject, often considered purely technical, actually conceals strategic choices that have a direct impact on the system's final performance.

Even if most large models currently offer context window sizes over several thousand tokens, it is not possible to have unit sizes that are too large, or it will be very difficult to find the content closest to the request. It should be noted that 100 tokens represent about 75 words in English, but fewer in French. These large windows are more useful, for example, to be able to use the conversation history, or to have many fragments in the context of the RAG, or to summarize long texts.

Context window size examples (as of mid-2025)

Several techniques can be applied in the segmentation phase:

Fixed-size segmentation

This is the most frequently used technique. It consists of cutting the text into segments of constant size. One of the main advantages of this method is its simplicity. It is very easy to implement, requires no complex linguistic analysis, and integrates easily into standard pipelines. In addition, it makes it possible to effortlessly respect the size limits imposed by language models while facilitating batch processing during the indexing phase.

This uniformity of the segments can also contribute to better stability during the generation of embeddings, as each segment is processed in a consistent format. This fixed size can be a set number of characters or tokens. The ideal size varies between 500 to 1000 characters, or between 200 and 600 tokens, or about 80 to 200 words. Often, an overlap of 10 to 20% is applied to avoid contextual losses (sliding window). Segments that are too short (less than 300 characters) can lack context and generate uninformative embedding. Segments that are too long (longer than 1500 characters) can drown out important information in noise, can include several topics, or simply exceed processing limits in some RAR architectures.

However, fixed cutting has several limitations. It tends to fragment content without taking into account the natural boundaries of language, paragraph structure, document, or unit of speech. This often results in a loss of context when sentences or ideas are cut off arbitrarily. This loss impairs the overall understanding of the segment, which can affect the relevance of the responses generated. In addition, this method can introduce redundancies when overlapping segments are used in an attempt to preserve context, or, conversely, create gaps and co-reference problems if no overlap is expected. In addition, it causes an overhead on the storage of cut segments. Finally, it can weigh down the index with uninformative segments, reducing the overall efficiency of the retrieval system.

Heuristic segmentation

Heuristic segmentation is often considered a superior approach to document slicing because it better preserves the meaning and coherence of content. Unlike fixed cuts, heuristic segmentation is based on rules that take into account the linguistic structure of the text. This approach leverages existing structural markers in the documentation (paragraphs, headings, paragraph breaks) to divide the text, allowing full semantic units to be retained.

This method has several advantages. First, it allows the meaning of the segments to be better preserved, which improves the understanding of the model when generating responses. Product segments are generally more coherent and carry a complete idea, which makes them more relevant during the information search phase. Avoiding arbitrarily cutting off a sentence or line of reasoning also reduces the risk of introducing noise or diluting useful information. In addition, this approach naturally adapts to the logical structure of documents, whether narrative, technical or legal.

However, heuristic segmentation requires more complex preprocessing, especially when documents contain heterogeneous or poorly formatted texts, producing inconsistent segments. In addition, the cutting can be disturbed by the presence of a bulleted list, a table, an image, or a graph. This approach must therefore rely on natural language processing tools to correctly detect sentences and sections. Its reliability depends on the initial quality of the text: poor punctuation or non-standard structure can reduce its effectiveness. In addition, it often produces segments of varying size, which can complicate their management in the database or their processing by certain models.

Segmentation by semantic cohesion

Algorithms such as TextTiling (available in the NLTK library) identify thematic transitions via lexical distribution analysis. This method better preserves design consistency, but is still sensitive to stylistic variations and requires intensive pre-processing.

This technique segments text into full thematic units through encodings to detect semantic breaks, ensuring that each segment encapsulates a standalone idea. Tools such as Unstructured AI, via intelligent segmentation, automate this segmentation while integrating non-textual elements (tables, diagrams).

Segmentation by LLM

LLMs like GPT-4 or Claude 3 can now be used to generate segments via specific prompts, capturing implicit logical links. Their ability to interpret context allows them to process technical or legal documents with increased accuracy.

The LumberChunker method proposes this approach in the study LumberChunker: Long-Form Narrative Document Segmentation:

• Semantic Analysis via LLM: Uses a language model to automatically detect “narrative breaks” — chapter transitions, changes in point of view, switching to a new theme, etc.
• Dynamic segmentation: Instead of rigid segmentation, segment sizes adjust according to content: one chapter can be a single coherent segment, while other sections are split more finely

LumberChunker Schematic (source: arXiv)

Meta-Chunking

Introduced at the end of last year by Renmin University of China (Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perception), the Meta-Chunking approach combines two strategies that seek to identify real logical and semantic breaks in the text, while compensating for the loss of context induced by segmentation.

To do this, the approach proposes perplexity slicing (PPL) and margin sampling slicing (MSP):

• Perplexity (PPL) Chunking: analyzes the perplexity distribution (a statistical measure used to assess the prediction quality of a language model on a sequence of text, which quantifies how “surprised” a model is by a sequence of words) to identify natural boundaries between concepts. By observing local variations in perplexity, the system can spot points where text becomes more predictable, which usually signals natural transitions.

“If the model is confident in its predictions, the perplexity is low”

• Margin Sampling Chunking: utilizes LLMs to evaluate the probabilities of segmentation through structured prompts, enabling dynamic adaptation to the context.

An adaptive fusion phase then regulates the granularity of the product segments, called “meta-chunks”. Once this segmentation has been carried out, the method proposes a complementary treatment that aims to fill in any information losses. Each segment is enriched via a multi-step rewriting phase: first by identifying the missing information, then selecting the information that is really useful, before integrating these elements in a coherent way into the text. At the same time, a double summary is generated: a global summary capturing the general meaning of the document, and a local summary specific to each segment. These two levels are then combined to enhance the completeness of the segment. This dynamic approach balances fine and coarse granularity by improving the efficiency of multi-hop queries by 45.8% in processing time. As with the LumberChunker approach, this process remains effective even when implemented with small language models such as Qwen2.5–3B, making it suitable for resource-constrained contexts.

Complex Document Management

Multimodal approaches aim to take into account all the content present in a document, whether textual or not. In particular, they integrate techniques for segmenting visual elements, such as images, tables, or diagrams, based on so-called layout-aware models, i.e., capable of taking into account the spatial structure of the document. In this context, the textual description of visual elements in the content stream plays a critical role in preserving overall semantic consistency. This operation consists of verbalizing content that is not directly textual but that provides important information, such as a table, a graph, or an illustration. For example, a data table can be transformed into a series of explicit and synthetic sentences, integrated into the surrounding text to make it easier to understand. Similarly, images can be interpreted and described using automatic captioning techniques (known as image captioning), which allows key information to be extracted and rendered in text form.

Dynamic adaptation

Systems like MoC (Mixtures of Text Chunking Learners) use machine learning to adjust segmentation strategies based on the type of document. This flexibility reduces errors by 32% on heterogeneous corpora.

Integration with knowledge graphs

The RAG Graph structures the content of segments into knowledge graphs, allowing complex relational queries (e.g., links between symptoms and treatments in medicine or links to several articles in a web page). The notion of segmentation disappears in favor of an atomic approach to information.

Document Structuring

To simplify the identification of linguistic units in a document and to avoid the need for the various complex segmentation techniques described above, it is recommended that documentation be restructured from the design stage. This restructuring aims to make content more algorithmically readable, navigable, interpretable, and modular, perhaps less for human readers than for language models.

The first adaptation is to write short, self-contained, and semantically coherent paragraphs. Each paragraph must carry a single complete idea, without linking several heterogeneous notions. A length of between 500 and 800 characters per paragraph is ideal for maintaining sufficient context while remaining easily manipulated by indexing or generation engines.

The systematic addition of explicit and hierarchical titles makes it possible to mark out the structure of the document. Creating a clear hierarchy of headers is a fundamental part of efficiency. The organization into logically ordered chapters, separated by explicit transitions between the thematic blocks, also makes it possible to mark breaks in context. Documents should use a logical progression of heading levels, avoiding skipping hierarchical levels. These headings should accurately reflect the content of the sections and subsections to guide the logical analysis of the document. Ideally, you should introduce each major part with a brief introduction that presents the main ideas discussed, accompanied by a mini-summary at the end that reformulates the key points. This type of controlled redundancy improves the semantic robustness of the document. Each segment must retain information about its parent section and its position in the global hierarchy.

It is also useful to include explicit reformulations or definitions of important terms in the body of the text at the time they appear, rather than relying solely on a final glossary that will very often be systematically ignored in the retrieval of information. This avoids co-reference or long-term contextual dependency issues and allows each text segment to have a standalone informative value. The use of pronouns should also be limited, which will avoid the use of coreference resolution techniques. References to external sources are interesting when using RAG techniques such as MetaRAG to take these links into account. They must therefore be identifiable. It is necessary to avoid references at the end of the document that are treated as a unit when they do not provide information locally.

When a document contains non-text elements such as tables, code portions, diagrams, or images, these must be accompanied by a text description embedded in the reading stream. When tables are essential, it is essential to avoid graphic effects and to favor simple structures without cell fusion. A table must be immediately preceded or followed by a summary of its contents.

Similarly, images should be captioned with sentences describing their role, meaning, or conclusions that can be drawn from them. This verbalization reduces the need for image captioning models or layout-aware systems. Rather than isolating tables and images, it’s best to integrate them into sections that explain their relevance and relationship to the surrounding content.

A complex layout, logos, headers, and footers, table of contents, revision notes, glossaries, and indexes must be avoided. These elements are decorative and informative for human readers, but induce noise in the segments. It is better to place them in the metadata of the document rather than inserting them in the body. Delete cover pages, legal notices, and signatures. Particular attention must be paid to the encoding of the characters (é instead of é). The lists must be standardized. Bold or capitalize important concepts in texts (depending on the format). Reduce syntactic complexity by avoiding long sentences with multiple nested subordinates.

Finally, a well-structured document should end with an overall summary that reformulates the main points covered, taking up the conclusions of each part. This summary layer makes it possible to create a bridge to the uses of information retrieval or assisted generation without the need for meta-chunking or semantic post-processing.

Documents should use a single-column structure because multi-column layouts can significantly reduce the accuracy of extraction and require fine-tuning. It is highly recommended to avoid compressed PDFs that can cause data distortions, multi-page tables that are difficult to process accurately, and inconsistent formatting that alternates between different layouts within the same document. These elements can compromise the quality of the extraction and introduce errors in the ingestion process.

Standardizing text formatting should be limited to the essentials. Excessive use of formatting, such as bold and colors, should be avoided, as these elements are usually not transcribed upon ingestion. It’s best to rely on a clear hierarchy of headings and a logical paragraph structure to organize content consistently. Nevertheless, the use of the Markdown format makes it possible to render several formatting elements, and it is relatively well handled by language models. The use of HTML portions nevertheless makes it possible to manage certain complex displays at the table level while promoting their interpretation.

You must add text navigation tags, such as [Business Rule: RB-42], that give information to the segment.

Markdown: a structuring, readable, and LLM-friendly format

The Markdown format is a lightweight markup language designed to structure text in a readable and logical way, for both humans and machines. It uses a simple syntax — characters such as # for headings, — or * for lists, and triple backticks for code — that allows the organization of a document to be described without resorting to complex tags as in HTML or XML.

This simplicity makes it particularly well-suited to the creation of technical documents that are structured, hierarchical, and easy to manipulate. Markdown’s structuring elements (headings, subheadings, tables, lists, code blocks) make the logic of the content explicit, which is of practical interest for automatic processing.

Language models interpret Markdown very well because its explicit syntax makes the document structure visible without complex parsing effort. For example, a ## Prerequisite or a ### Steps indicates an organizational intent that the template can exploit to better contextualize an answer. In addition, sections are easier to segment and index, as they are delimited in a standardized way. Unlike PDF or Word documents, where visual cues are ambiguous or lost during conversion, Markdown keeps its structural cues intact and legible. It is this transparency of format that makes it an excellent choice for RAG systems: documents are simpler to parse, structure, and interpret by LLMs, which directly improves the quality of the responses generated.

Hierarchical directory architecture

Enrichment of segments with contextual metadata significantly improves the accuracy of information retrieval (MetaRAG approach). Each segment should include information about its source, its position in the original document, and semantic keywords that make indexing easier.

The organization of directories must maximize the efficiency of retrieval, regardless of human navigation needs. The directory hierarchy should reflect the key dimensions used during the recovery phase.

Thus, a tree structure that classifies documents at several levels is optimal:

• Level 1: Semantic domain (e.g. ‘/contracts/’, ‘/rapports_techniques/’, ‘/communications/’)
• Level 2: Temporal granularity (e.g. ‘/2023/’, ‘/Q2_2024/’)
• Level 3: Named entities (e.g., ‘/client_X/’, ‘/projet_Y/’)

This structure allows for automatic indexing of contextual metadata during ingestion. Each hierarchical level generates complementary embeddings that enrich the context of the segments.

Document titles should be specific and descriptive, favoring wording like “Two-factor authentication setup” rather than generic terms like “Security.” The file names follow a format that must be usable by the machine, such as:

[doc_type]-[creation_date]-[uniqueid][keywords].extension

Example: ‘contract-20230512–789AC_renew-clientX.pdf’

This pattern facilitates the automatic extraction of metadata via regular expressions during preprocessing.

Documents can also be grouped according to their predictable internal structure, business functions, or use case:

Directory tree example

A symlink system creates multiple views without duplication:

• /contracts/client_X/ -> /2023/customers/X/contracts
• /reports/client_X/ -> /2023/clients/X/reports

Establishing a consistent metadata system effectively complements structural categorization. This metadata should include information about the document type, scope, creation date, and relevant keywords. It may be useful to add an “index.yaml” file in each directory to list the major themes, associated files, internal sections, or specific processing rules for each file. This approach allows the RAG system to better understand the context and relevance of each document during retrieval processes:

YAML metadata example

This metadata can feed filters during search.

The use of a department structure (general management, human resources, finance and accounting, IT, marketing, sales, production and operations, purchasing, etc.) can be particularly effective in complex organizational environments. Each department has its main folder, with subfolders corresponding to specific tasks and frequent topics. This organization allows the RAG system to maintain a clear separation between the different areas of expertise while maintaining the necessary flexibility for cross-functional queries.

Indexing frequencies

The index must reflect the company’s up-to-date information. This automation is especially important for dynamic environments where documents are frequently modified. Changes must be easy for the ingestion system to detect. A system that only provides the delta since the last visit avoids having to go through the whole tree again.

There are several possible strategies at the document level:

• Comprehensive update. Each document is fully updated, involving the removal of the existing index and the complete crawling of the document regardless of its size. The process is launched either regularly, depending on the frequency of document updates: monthly, daily, etc., or on a trigger related to the addition of a new document or an update. It is possible to keep all versions so that you can go back or compare different versions.
• Partial update. Only the modified parts of the document are analyzed and stored. This implies being able to detect the modified parts of a document (add, modify, delete).

Designing an ingestion pipeline requires anticipating growth in data volumes and diversification of information sources.

Conclusion

Optimizing document organization for RAG systems is a strategic investment that directly impacts the quality and relevance of the generated responses. Implementing a clear hierarchical structure, along with appropriate rewriting techniques and simplified segmentation strategies, is foundational to a successful RAG system. Paying attention to organizational details, from file naming to content structuring, determines the system’s ability to effectively understand and retrieve relevant information.

Organizations that invest in careful preparation of their documents reap significant benefits in terms of retrieval accuracy and quality of responses. Adopting standardized practices for formatting, categorizing, and segmenting documents creates an optimal environment for automatic ingestion and semantic search. This systematic approach also helps to maintain the consistency and quality of the system as the database evolves and expands over time.

From a perspective where a more “graphic” presentation is required for users, it is possible to set up tools to facilitate the viewing and navigation of documentary content. Automatic generators of table of contents, indexes, and glossaries can structure long documents and enhance navigation. Similarly, rich markdown extensions or formats such as rich HTML can be used to visually display the hierarchy of information. These tools, by making document units more visible, logical, and connected, simplify the user experience, without compromising the indexing and automatic segmentation of content by RAG systems.

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!