🚀 Building Production-Ready LLM Systems with LangChain, RAG, ChromaDB, and Neo4j Knowledge Graphs

Modern AI systems are no longer just “LLM + prompt”. Production-grade systems require:

• Persistent memory (Vector DB)
• Structured reasoning (Knowledge Graphs)
• Retrieval pipelines (RAG)
• Deduplication + versioning
• Scalable ingestion pipelines
• Observability + control

This article walks through a real production architecture combining:

🔷 LLM + LangChain 🔷 RAG (Retrieval Augmented Generation) 🔷 ChromaDB (Vector Store) 🔷 Neo4j (Knowledge Graph) 🔷 Dedup-aware ingestion pipeline 🔷 Hybrid retrieval strategy

1. High-Level Architecture

                ┌──────────────┐
                │   Documents   │
                └──────┬───────┘
                       │
            ┌──────────▼──────────┐
            │  Ingestion Pipeline  │
            │ (Chunk + Clean + ID) │
            └──────────┬──────────┘
                       │
        ┌──────────────┴──────────────┐
        │                             │
┌───────▼────────┐        ┌──────────▼─────────┐
│  ChromaDB       │        │    Neo4j KG        │
│ (Vector Search) │        │ (Entities + Graph) │
└───────┬────────┘        └──────────┬─────────┘
        │                             │
        └──────────────┬──────────────┘
                       ▼
            ┌─────────────────────┐
            │ LangChain Retriever │
            └─────────┬───────────┘
                      ▼
                ┌──────────┐
                │   LLM    │
                └──────────┘
                      ▼
                Final Answer

2. Core Components

2.1 ChromaDB (Vector Database)

Used for:

• Semantic similarity search
• Chunk-level retrieval
• Fast ANN search

import chromadb
from chromadb.config import Settings

chroma_client = chromadb.PersistentClient(
    path="./chroma_store"
)

collection = chroma_client.get_or_create_collection(
    name="docs",
    metadata={"hnsw:space": "cosine"}
)

2.2 Neo4j Knowledge Graph

Used for:

• Entity relationships
• Structured reasoning
• Multi-hop queries

from langchain.embeddings import OpenAIEmbeddings

embedding_model = OpenAIEmbeddings()

def embed_chunks(chunks):
    return embedding_model.embed_documents(chunks)

Step 3: Deduplication Strategy (VERY IMPORTANT)

Problem:

You load:

• Document A (already exists)
• Document B (contains overlapping sections)

You risk:

• Duplicate embeddings
• Polluted retrieval
• Biased ranking

Production Solution: Multi-layer Dedup

✔ Layer 1: Hash-based chunk dedup

import hashlib

def chunk_hash(text):
    return hashlib.md5(text.encode()).hexdigest()

Store:

• chunk_id
• hash
• document_id

Layer 2: Semantic dedup (vector similarity threshold)

If cosine similarity > 0.95 → treat as duplicate

def is_duplicate(new_embedding, existing_embeddings, threshold=0.95):
    # pseudo logic
    return max_similarity > threshold

✔ Layer 3: Graph-level dedup (Neo4j)

If entity already exists → merge relationships

4. Storing Data in ChromaDB

def store_in_chroma(chunks, embeddings, doc_id):
    for i, (chunk, emb) in enumerate(zip(chunks, embeddings)):
        chunk_id = f"{doc_id}_{i}"
        hash_id = chunk_hash(chunk)

        collection.add(
            ids=[chunk_id],
            embeddings=[emb],
            documents=[chunk],
            metadatas=[{
                "doc_id": doc_id,
                "hash": hash_id
            }]
        )

5. Knowledge Graph Extraction (Neo4j)

We extract:

• Entities
• Relationships

Example:

“Tesla was founded by Elon Musk in 2003”

Graph:

(Elon Musk) -[FOUNDED]-> (Tesla)

Extraction with LLM

from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini")

prompt = """
Extract entities and relationships from text:
Return JSON format:
{
  "nodes": [],
  "edges": []
}

Text: {text}
"""

Store in Neo4j

def store_graph(nodes, edges):
    with driver.session() as session:
        for node in nodes:
            session.run(
                "MERGE (n:Entity {name: $name})",
                name=node
            )

        for edge in edges:
            session.run("""
                MATCH (a:Entity {name: $from})
                MATCH (b:Entity {name: $to})
                MERGE (a)-[:RELATION {type: $type}]->(b)
            """, edge)

6. Hybrid Retrieval (Vector + Graph)

Step 1: Vector retrieval

results = collection.query(
    query_embeddings=[query_embedding],
    n_results=5
)

Step 2: Graph expansion

If query contains entity → expand neighbors

MATCH (n:Entity {name: "Tesla"})-[]-(connected)
RETURN connected

Step 3: Merge results

LangChain retriever:

from langchain.schema import Document

def hybrid_retrieval(query):
    vector_docs = vector_search(query)
    graph_docs = graph_search(query)

    return vector_docs + graph_docs

7. LangChain RAG Pipeline

from langchain.chains import RetrievalQA

qa = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=custom_retriever,
    return_source_documents=True
)

response = qa.run("What is Tesla?")

8. Production Considerations

8.1 Scalability

• Use batch embedding
• Async ingestion pipeline
• Queue system (Kafka / Redis Queue)

8.2 Observability

Track:

• retrieval latency
• embedding cost
• duplicate rate
• graph growth

8.3 Versioning

Each document should have:

{
  "doc_id": "abc",
  "version": 3
}

8.4 Cost optimization

• Cache embeddings
• Deduplicate aggressively
• Hybrid retrieval reduces LLM tokens

9. Real Scenario: Old + New Document Ingestion

Scenario

You already have:

📄 Document A:

• “Tesla was founded in 2003…”

Now ingest:

📄 Document B:

• “Elon Musk founded Tesla in 2003…”

What happens?

Vector DB:

• detects near-identical chunks → avoids storing duplicates

Neo4j:

• merges:

(Elon Musk) -[FOUNDED]-> (Tesla)

If relation already exists:

• updates metadata instead of duplicating edge

Result:

| Layer    | Behavior                           |
| -------- | ---------------------------------- |
| ChromaDB | avoids duplicate embeddings        |
| Neo4j    | merges entity relationships        |
| RAG      | improves recall without redundancy |

10. Common Production Pitfalls

• No chunk dedup → vector pollution
• No graph normalization → duplicate entities
• No metadata tracking → untraceable answers
• Too large chunks → poor retrieval
• No hybrid retrieval → shallow reasoning