From Traditional RAG to GraphRAG: Building Smarter AI Systems with Coarse-Grained and Fine-Grained Knowledge Graphs

How Neo4j, LangChain, and Large Language Models combine vector search and graph reasoning for next-generation Retrieval-Augmented Generation

Traditional RAG retrieves similar documents. GraphRAG retrieves connected knowledge.

As enterprise AI applications become more complex, simply retrieving the top-k document chunks is no longer enough. Important information is often distributed across multiple sections of a document or spread across interconnected business entities. This is where Knowledge Graphs transform Retrieval-Augmented Generation (RAG), enabling richer context, multi-hop reasoning, and more accurate answers.

In this article, we’ll explore the evolution of RAG — from traditional vector search to Coarse-Grained and Fine-Grained Knowledge Graphs — and learn when to use each architecture.

Introduction

Retrieval-Augmented Generation (RAG) has become one of the most widely adopted architectures for building AI applications on top of Large Language Models (LLMs). Instead of relying solely on the model’s pre-trained knowledge, RAG retrieves relevant information from an external knowledge base and provides it as context before generating a response.

This approach significantly improves factual accuracy, reduces hallucinations, and allows LLMs to answer questions over private or domain-specific data.

A traditional RAG pipeline is straightforward:

Most implementations use vector databases such as Neo4j Vector, FAISS, Pinecone, Chroma, Milvus, or Weaviate to perform similarity search.

A simple LangChain example looks like this:

docs = vector_store.similarity_search(
    query=question,
    k=3
)

For many applications, this works remarkably well.

However, when documents become larger and more complex — or when working with structured enterprise systems — traditional RAG begins to reveal its limitations.

The Limitation of Traditional RAG

During one of my recent projects, I was building an AI assistant over insurance policy documents.

One section of the document explained the cancellation criteria:

Cancellation Policy

Policy cancelled within the first month...

Several pages later, another section contained the refund table:

1-Year Policy → 75%

2-Year Policy → 87.5%

3-Year Policy → 92%

Although both sections were logically related, a vector search frequently retrieved only one of them. As a result, the LLM generated incomplete or inaccurate responses because it lacked the full context.

The issue wasn’t the language model — it was the retrieval process.

This observation led to a simple question:

Can we connect related pieces of information so retrieval doesn’t stop at a single chunk?

That idea naturally leads to GraphRAG.

What is GraphRAG?

GraphRAG extends traditional Retrieval-Augmented Generation by incorporating graph structures into the retrieval process.

Instead of retrieving isolated chunks of information, GraphRAG retrieves connected knowledge, allowing the system to gather richer context before passing it to the language model.

Depending on the nature of your data, GraphRAG can be implemented in two distinct ways:

• Coarse-Grained Knowledge Graphs for unstructured documents.
• Fine-Grained Knowledge Graphs for structured enterprise data.

Let’s explore each approach.

Part 1: Coarse-Grained Knowledge Graph RAG

What is a Coarse-Grained Knowledge Graph?

Traditional knowledge graphs model entities and relationships.

For example:

Customer ── OWNS ──► Policy
Policy ── COVERS ──► Hospital
Doctor ── TREATS ──► Patient

Building such graphs requires entity extraction, relation extraction, ontology design, and entity resolution, making the process both complex and time-consuming.

For many document-centric applications, however, this level of complexity isn’t necessary.

Instead, we can model the document itself as a graph.

Each document chunk becomes a node:

(:Chunk)

Semantically similar chunks are connected using a SIMILAR_TO relationship:

Rather than connecting entities, we connect related pieces of information.

This approach is called a Coarse-Grained Knowledge Graph.

Building a Coarse-Grained Knowledge Graph

The ingestion pipeline consists of five straightforward steps:

1. Load PDF documents.
2. Split them into chunks.
3. Generate embeddings.
4. Store chunks in Neo4j.
5. Connect semantically similar chunks.

Step 1 — Load and Chunk Documents

from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = PyPDFLoader("insurance_policy.pdf")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)

chunks = splitter.split_documents(documents)

Step 2 — Generate Embeddings

from langchain_openai import OpenAIEmbeddings

embedding_model = OpenAIEmbeddings(
    model="text-embedding-3-small"
)

Step 3 — Store Chunks in Neo4j

Neo4jVector.from_documents(
    documents=chunks,
    embedding=embedding_model,
    ...
)

Each graph node stores:

• Chunk text
• Embedding vector
• Metadata

Step 4 — Connect Similar Chunks

MATCH (c:Chunk)
CALL db.index.vector.queryNodes(
'graph_store',
20,
c.embedding
)
YIELD node AS other, score
WHERE score >= 0.75
MERGE (c)-[:SIMILAR_TO {score: score}]->(other)

Neo4j now stores both vector embeddings and graph relationships within the same database.

Retrieval with Graph Expansion

Instead of stopping after vector search, retrieval now happens in two stages:

1. Retrieve the most relevant seed chunks using vector search.
2. Traverse SIMILAR_TO relationships to collect neighboring chunks.

This graph expansion enriches the retrieved context before sending it to the LLM, significantly improving answers for long documents where related information spans multiple sections.

Best Use Cases

Coarse-Grained Knowledge Graphs work particularly well for:

• Insurance policies
• Legal contracts
• Technical manuals
• Compliance documents
• Research papers
• Multi-page reports

Part 2: Fine-Grained Knowledge Graph RAG

While Coarse-Grained Knowledge Graphs are ideal for unstructured documents, structured enterprise systems require a fundamentally different approach.

Instead of modeling document chunks, we model business entities.

What is a Fine-Grained Knowledge Graph?

A Fine-Grained Knowledge Graph represents every business object as an independent node connected through explicit business relationships.

For a procurement system, the graph might include:

• Purchase Orders
• Purchase Requests
• Contracts
• Invoices
• Suppliers

Relationships represent the actual business workflow:

Unlike Coarse-Grained graphs, these relationships are not inferred — they already exist within the enterprise domain.

Fine-Grained GraphRAG Architecture

A production-ready enterprise GraphRAG pipeline consists of several stages:

1. User question
2. Guardrails and input validation
3. Fuzzy entity search
4. Natural Language → Cypher generation
5. Neo4j graph execution
6. Structured graph retrieval
7. Semantic vector retrieval
8. Hybrid context fusion
9. Final answer generation

Why Hybrid Retrieval?

Enterprise data contains both structured and unstructured information.

Graph traversal retrieves factual relationships:

• Purchase Orders
• Suppliers
• Contracts
• Invoices

Vector search retrieves semantic descriptions:

• Contract clauses
• Invoice notes
• Purchase descriptions
• Business comments

Combining both creates a richer context for the language model.

Advantages of Fine-Grained Knowledge Graphs

Fine-Grained GraphRAG offers several enterprise benefits:

• Preserves business relationships
• Supports accurate multi-hop reasoning
• Enables explainable query results
• Reduces hallucinations with schema-aware retrieval
• Supports typo-tolerant search using Lucene
• Combines graph and vector retrieval
• Integrates naturally with Guardrails AI
• Improves numerical accuracy by separating retrieval from computation

Coarse-Grained vs Fine-Grained Knowledge Graphs

| Feature        | Coarse-Grained KG        | Fine-Grained KG                |
| -------------- | ------------------------ | ------------------------------ |
| Node           | Document Chunk           | Business Entity                |
| Relationships  | Semantic Similarity      | Business Relationships         |
| Graph Creation | Automatic                | Schema Driven                  |
| Retrieval      | Vector + Graph Expansion | Cypher + Multi-Hop Traversal   |
| Best For       | PDFs, Manuals, Policies  | ERP, CRM, Finance, Procurement |
| Complexity     | Low                      | High                           |
| Explainability | Medium                   | Very High                      |

Which One Should You Choose?

Choose Coarse-Grained Knowledge Graphs when working with large collections of unstructured documents such as PDFs, manuals, legal contracts, or insurance policies. They are simple to implement, require no ontology design, and significantly improve document retrieval through graph expansion.

Choose Fine-Grained Knowledge Graphs when your data already contains well-defined business entities and relationships, such as procurement systems, ERP platforms, CRM applications, finance, healthcare, or supply chain systems. They provide accurate multi-hop reasoning, explainable retrieval, and enterprise-grade GraphRAG capabilities.

Conclusion

GraphRAG represents the next evolution of Retrieval-Augmented Generation.

Traditional vector search retrieves documents based on semantic similarity, but it often lacks the relational context needed to answer complex questions.

A Coarse-Grained Knowledge Graph addresses this challenge for unstructured documents by connecting semantically similar chunks, enabling richer retrieval with minimal implementation effort.

A Fine-Grained Knowledge Graph takes a different approach by modeling real business entities and their relationships. Combined with schema-aware Cypher generation, hybrid graph and vector retrieval, and multi-hop reasoning, it forms a powerful foundation for enterprise AI systems.

Rather than viewing these architectures as competitors, think of them as complementary solutions. The right choice depends on the nature of your data:

• Unstructured documents → Coarse-Grained GraphRAG
• Structured enterprise data → Fine-Grained GraphRAG

As AI applications continue to evolve, combining vector search, knowledge graphs, and large language models will play a central role in building systems that are more accurate, explainable, and context-aware.

About the Author

I’m an AI Engineer passionate about building production-ready Generative AI, GraphRAG, Knowledge Graph, and LLM solutions using technologies such as Neo4j, LangChain, Azure OpenAI, and Python. I enjoy exploring practical architectures that bridge the gap between research and real-world enterprise applications.

If you enjoyed this article, feel free to connect with me on LinkedIn and follow my Medium profile for more content on GraphRAG, LLMs, Knowledge Graphs, Agentic AI, and Enterprise AI Architecture.