Yes, another RAG post: but this time let’s open the box a little bit

Retrieval-Augmented Generation is everywhere now. There are already many good ways to build RAG without touching most of the low-level pieces. And that's great!

But, as you may already know, abstraction is only helpful when you understand, at least a little bit, what is being abstracted. If all we see is a nice function call or a button in an interface, we may miss the decisions that actually determine whether the system works. So let’s open this box (a little bit).

We do not need to reimplement everything from scratch. Please don’t. But RAG failures are often hidden behind nice abstractions. You get a clean answer from a well-controlled prompt, a few retrieved chunks, maybe even a citation, and everything looks fine.

Until you put it in the hands of real users. Then you realize the model was confident, the chunks were “semantically similar”, the output looked polished, but the answer was still not grounded in the document you actually cared about. It fails by sounding reasonable, which is much more annoying.

In this post (probably not the first on this topic, but let’s see), I want to build a very simple RAG pipeline from a slightly lower level. If that makes sense to you, let’s go :)

A mental model for understating this thing

The term RAG comes from a quite interesting paper from a few years ago (2020). The important idea is not “vector database + LLM”. The important idea is the combination of two kinds of memory:

1. Parametric memory: knowledge stored in the model weights.
2. Non-parametric memory: explicit external knowledge that can be retrieved, inspected, replaced, versioned, and cited.

That distinction matters. An LLM’s parametric memory is powerful, but it is not a database. It is compressed, implicit, hard to patch, and not naturally source-aware. External memory is less magical, but much easier to govern. You can update a PDF, re-index a policy, remove a stale page, add metadata, filter by date, and ask “where did this answer come from?”

RAG is useful because it gives the model a controlled reading list at inference time.

RAG is not the model knowing more. RAG is the application bringing evidence to the model at the right time.

That difference sounds small, but it changes how you design the system. If the model “knows”, you tend to trust the answer. If the application brings evidence, you can inspect the evidence, measure retrieval, cite sources, filter by permissions, and decide what to do when the evidence is weak.

Let’s try a classical metaphor here: the corpus is the library, the retriever is the librarian, the prompt is the reading desk, the generator is the writer.

If the librarian brings the wrong books, the writer may still produce a beautiful paragraph. It will just be beautifully unsupported.

This is also why I do not like describing RAG as “the LLM connected to your data” without more context. The connection is not enough. The system needs a retrieval policy, metadata, relevance thresholds, citations, and a refusal path when the evidence is not there.

RAG does not automatically make answers correct, it can still fail in at least four very ordinary ways:

1. It retrieves the wrong passages.
2. It retrieves the right passage plus enough noise to distract the model.
3. It retrieves contradictory passages and never resolves the conflict.
4. It retrieves good evidence, but the generation step ignores it.

So let's think of RAG less as a feature and more as a contract:

Retrieval proposes evidence. Augmentation defines the allowed context. Generation is constrained to that context.

The hard engineering lives in enforcing that contract.

Where to start?

Let’s treat RAG as a small information architecture problem:

• What is the model allowed to know from its parameters?
• What should be retrieved from an external memory?
• How do we decide that a chunk is good enough?
• How do we expose retrieval as a tool instead of blindly injecting context into every prompt?

In this example, we will use MongoDB as the vector database. The same general architecture could also be implemented with other options, such as PostgreSQL with pgvector, Qdrant, Weaviate, Pinecone, or Elasticsearch.

The goal here is not to build the most impressive setup. It’s to build something you can understand, debug, and improve.

The pipeline

A practical RAG pipeline usually has two phases:

Indexing time

1. Select the corpus.
2. Parse the documents.
3. Split them into chunks.
4. Generate embeddings.
5. Store the chunks, embeddings, and metadata.
6. Build an index for retrieval.

Query time

1. Receive a user question.
2. Embed the question with the same embedding model.
3. Retrieve candidate chunks.
4. Optionally filter, rerank, deduplicate, or validate them.
5. Build a grounded prompt.
6. Generate an answer with citations.

The first non-obvious lesson is that chunking is not a boring preprocessing step. Chunking is where you decide the unit of meaning your retriever can recover.

If chunks are too small, you retrieve isolated fragments without enough context. If they are too large, similarity gets diluted and the model receives unrelated text. Overlap helps, but overlap also increases duplication. There is no universal chunk size. A legal contract, a short text document, a table-heavy PDF, and a codebase may require different splitting strategies.

Setup

First, let’s start clean. I recommend creating a Python virtual environment to keep dependencies isolated:

python3 -m venv .venv
source .venv/bin/activate
# now install the dependencies:
pip3 install pymongo sentence-transformers langchain-community langchain-text-splitters pypdf google-genai einops

We'll use:

1. A public NASA technical PDF (Systems Engineering Handbook) as an example for our document base. We will load it, split it into moderately sized chunks, and preserve page-level metadata (that metadata is not decoration; it is what makes citations possible).
2. nomic-ai/nomic-embed-text-v1 for embeddings.
3. MongoDB Atlas Vector Search for the vector index.

To create your database:

• Go to MongoDB Atlas and create an account (or sign in).
• Click Create a Cluster. Select the Free tier and choose a cluster name (rag_db).
• In the security popup, add your current IP address and create a database (username + password).
• Then, in Connect → Drivers, copy the connection string (URI). You'need to set it in yout env as MONGODB_URI. It will look like this: mongodb+srv://<user>:<pwd>@<cluster_url>/?appName=rag_db

Ingestion phase

Here is an ingestion script (take a look at its anatomy!). It loads the PDF, splits it into chunks, embeds each chunk, stores everything in MongoDB, and creates a vector search index.

import os
import time
from hashlib import sha256

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from pymongo import MongoClient, UpdateOne
from pymongo.operations import SearchIndexModel
from sentence_transformers import SentenceTransformer

# Public technical PDF. It has sections, definitions, process descriptions,
# and enough structure to test retrieval.
PDF_URL = os.getenv(
    "PDF_URL",
    "https://www.nasa.gov/wp-content/uploads/2018/09/nasa_systems_engineering_handbook_0.pdf",
)

MONGODB_URI = os.environ["MONGODB_URI"] # Use the URI you retrieve from Atlas
DB_NAME = os.getenv("MONGODB_DB", "rag_db")
COLLECTION_NAME = os.getenv("MONGODB_COLLECTION", "document_chunks")
INDEX_NAME = os.getenv("MONGODB_VECTOR_INDEX", "vector_index")
EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "nomic-ai/nomic-embed-text-v1")

def stable_chunk_id(source: str, page: int, chunk_text: str) -> str:
    # A stable ID avoids inserting duplicate chunks when the ingestor is run again.
    raw = f"{source}:{page}:{chunk_text}".encode("utf-8")
    return sha256(raw).hexdigest()

# Load the PDF and preserve page-level metadata.
# This matters later because citations need to point back to where the text came from.
pages = PyPDFLoader(PDF_URL).load()

# Chunking is not just preprocessing.
# This is where we define the unit of meaning the retriever will be able to recover.
splitter = RecursiveCharacterTextSplitter(
    chunk_size=700,
    chunk_overlap=120,
    separators=["\n\n", "\n", ". ", " ", ""],
)

chunks = splitter.split_documents(pages)

encoder = SentenceTransformer(EMBEDDING_MODEL, trust_remote_code=True)
dimension = encoder.get_sentence_embedding_dimension()

client = MongoClient(MONGODB_URI)
collection = client[DB_NAME][COLLECTION_NAME]

operations = []

for chunk in chunks:
    page = int(chunk.metadata.get("page", 0)) + 1
    source = chunk.metadata.get("source", PDF_URL)
    text = chunk.page_content.strip()

    if not text:
        continue

    # For clarity, this example embeds one chunk at a time.
    # In production, batch this step for better performance.
    embedding = encoder.encode(text).tolist()

    operations.append(
        UpdateOne(
            {"chunk_id": stable_chunk_id(source, page, text)},
            {
                "$set": {
                    "text": text,
                    "page": page,
                    "source": source,
                    "embedding_model": EMBEDDING_MODEL,
                    "embedding": embedding,
                }
            },
            upsert=True,
        )
    )

if operations:
    collection.bulk_write(operations)

# Create a MongoDB Atlas Vector Search index over the embedding field.
# The source and page filters are useful later for citation and scoped retrieval.
search_index_model = SearchIndexModel(
    definition={
        "fields": [
            {
                "type": "vector",
                "path": "embedding",
                "numDimensions": dimension,
                "similarity": "cosine",
            },
            {"type": "filter", "path": "source"},
            {"type": "filter", "path": "page"},
        ]
    },
    name=INDEX_NAME,
    type="vectorSearch",
)

collection.create_search_index(model=search_index_model)

# Atlas may take a little while before the index is queryable.
while True:
    matches = list(collection.list_search_indexes(INDEX_NAME))
    if matches and matches[0].get("queryable") is True:
        break
    time.sleep(5)

print(f"Ingested {len(operations)} chunks into {DB_NAME}.{COLLECTION_NAME}")

A few details here are easy to skip and painful later:

• I keep source and page next to the text.
• I store the embedding_model, because changing embedding models means changing the geometry of your vector space.
• I use a stable chunk id, so re-running ingestion does not blindly duplicate the corpus.
• I create filter fields in the index, because RAG almost always needs metadata constraints.

Now if you open the database, you should see something like this:

Querying the vector index

At query time, we embed the user question and search for nearby chunks. (don't worry, we'll execute the full query script later)

def search_manual(question: str, limit: int = 5) -> dict[str, Any]:
    query_vector = encoder.encode(question).tolist()

    pipeline = [
        {
            "$vectorSearch": {
                "index": INDEX_NAME,
                "path": "embedding",
                "queryVector": query_vector,
                "numCandidates": 100,
                "limit": limit,
            }
        },
        {
            "$project": {
                "_id": 0,
                "text": 1,
                "page": 1,
                "source": 1,
                "score": {"$meta": "vectorSearchScore"},
            }
        },
    ]

    return {"passages": list(collection.aggregate(pipeline))}

The retrieval flow is: query → approximate search → candidate set (numCandidates) → exact scoring → top-k (limit) → return

numCandidates controls how many potential matches the vector search considers internally before selecting the final results. Because the search is approximate (ANN), it does not scan the entire dataset; it explores part of the vector space and builds a shortlist. From this shortlist (numCandidates), the system then picks the top results defined by limit. If numCandidates is too small, relevant chunks may never be considered and therefore cannot appear in the final results. Increasing it improves recall (higher chance of retrieving the right passages), at the cost of additional latency.

Notice the separation of responsibilities.

The retriever does not answer the question. It returns evidence. The generator (we'll implement it next) won't search the database directly. It will receive evidence through a defined tool.

This separation is what lets us test the retrieval layer independently. Before blaming the LLM, we can inspect the chunks. If the right passage is not in the top results, we do not have a generation problem yet. You have a retrieval problem.

Let a Generative AI decide when to retrieve

In many tutorials and setups out there, the application always retrieves context before calling the LLM. It’s not the only pattern, and I’ll take a bit of poetic license here.

Sometimes you want the model to decide whether a tool is needed. This is especially useful when your assistant can handle both general conversation and document-grounded questions. The current Google's Gemini Python SDK supports automatic function calling, which will come in handy here: you pass Python functions as tools, and the SDK can handle the function-call loop for you.

To make it work you'll need an API key from Google AI Studio (GOOGLE_API_KEY). You can also change GEMINI_MODEL as needed.

Here is the full query script:

import os
from typing import Any

from google import genai
from google.genai import types
from pymongo import MongoClient
from sentence_transformers import SentenceTransformer

MONGODB_URI = os.environ["MONGODB_URI"]
DB_NAME = os.getenv("MONGODB_DB", "rag_db")
COLLECTION_NAME = os.getenv("MONGODB_COLLECTION", "document_chunks")
INDEX_NAME = os.getenv("MONGODB_VECTOR_INDEX", "vector_index")
EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "nomic-ai/nomic-embed-text-v1")
GEMINI_MODEL = os.getenv("GEMINI_MODEL", "gemini-3-flash-preview")
GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY")

mongo = MongoClient(MONGODB_URI)
collection = mongo[DB_NAME][COLLECTION_NAME]

# Use the same embedding model used during ingestion.
encoder = SentenceTransformer(EMBEDDING_MODEL, trust_remote_code=True)

client = genai.Client(api_key=GOOGLE_API_KEY)

def search_manual(question: str, limit: int = 5) -> dict[str, Any]:
    """Searches for passages relevant to a user question.

    Args:
        question: A natural-language question.
        limit: Maximum number of passages to retrieve (recommended: 3 to 8).

    Returns:
        A dictionary containing the top-k retrieved passages,
        including text, source page, source URL, and similarity score.
    """
    print(f"[DEBUG][search_manual] called")
    print(f"[DEBUG][search_manual] question: {question}")
    print(f"[DEBUG][search_manual] limit: {limit}")

    query_vector = encoder.encode(question).tolist()

    pipeline = [
        {
            "$vectorSearch": {
                "index": INDEX_NAME,
                "path": "embedding",
                "queryVector": query_vector,
                # Retrieve more candidates than the final limit.
                # This usually improves the quality of the top-k results.
                "numCandidates": 100,
                "limit": limit,
            }
        },
        {
            "$project": {
                "_id": 0,
                "text": 1,
                "page": 1,
                "source": 1,
                "score": {"$meta": "vectorSearchScore"},
            }
        },
    ]

    results = list(collection.aggregate(pipeline))
    print(f"[DEBUG][search_manual] retrieved: {len(results)} passages")
    return {"passages": list(collection.aggregate(pipeline))}

SYSTEM_INSTRUCTION = """
You answer questions using only the passages returned by the search_manual tool.

Rules:
- Always call search_manual before answering.
- Do not use outside knowledge.
- If the retrieved passages do not contain enough evidence, say that the document does not provide enough information.
- Cite page numbers using this format: (p. X).
- Keep the answer concise and grounded in the retrieved text.
"""

def ask(question: str) -> str:
    response = client.models.generate_content(
        model=GEMINI_MODEL,
        contents=question,
        config=types.GenerateContentConfig(
            system_instruction=SYSTEM_INSTRUCTION,
            tools=[search_manual],
            temperature=0.2,
        ),
    )

    return response.text or ""

if __name__ == "__main__":
    question = input("> ")
    print(ask(question))

The model has a tool, and the tool exposes retrieval. We are not building an autonomous agent. We are letting the model call a constrained function that searches a constrained corpus. That constraint is a feature.

Run it with a question like “What is systems engineering?”. The debug logs should show the tool being invoked and the retrieved passages, followed by a (hopefully good) contextualized answer with citations.

[DEBUG][search_manual] called
[DEBUG][search_manual] question: What is systems engineering?
[DEBUG][search_manual] limit: 5
[DEBUG][search_manual] retrieved: 5 passages

> Systems engineering is a disciplined, holistic, and integrative engineering approach used for the development, operation, and maintenance of systems throughout their life cycle (p. 13, 203). It is characterized as both a "logical way of thinking" and the "art and science of developing an operable system" that meets requirements within often conflicting constraints (p. 13).

Key aspects of systems engineering include:
*   **Broad Perspective:** It uses a crosscutting "big picture" view rather than a single-discipline view to ensure the design meets requirements ("getting the design right") and fulfills stakeholder expectations ("getting the right design") (p. 14).
*   **Methodology:** It is a quantifiable, recursive, iterative, and repeatable process (p. 203).
*   **Tradeoffs:** It involves balancing organizational, cost, and technical interactions through tradeoffs and compromises (p. 14).
*   **Integration:** It integrates contributions from various disciplines—such as structural, electrical, and human factors engineering—into a cohesive whole (p. 13).

Going beyond semantic search

Vector search gives you candidate evidence. RAG adds a generation step that composes an answer from that evidence. But a serious RAG system usually needs more than top-k vector search:

• Hybrid retrieval: combine dense search with lexical methods (e.g., BM25). Vector similarity alone is weak for exact terms, IDs, or codes.
• Metadata filtering: restrict by version, date, product, tenant, or permissions.
• Reranking: retrieve broadly, then reorder with a cross-encoder or reranker.
• Deduplication: avoid sending overlapping chunks (a side effect of chunk overlap).
• Contradiction handling: prefer newer or authoritative sources when passages disagree.
• SLM gating / discriminator: use a smaller model before and after generation to control cost and quality, first as a router (decide if retrieval or a full LLM call is needed, or if the SLM can answer directly), and then as a verifier (check grounding, citation alignment, and hallucinations); only escalate to the larger model when necessary.
• Multimodal retrieval: many documents are not purely textual. Images, charts, and tables often carry the core information. Text-only embeddings will miss or distort this. Options include: extracting captions and surrounding text, generating image embeddings (e.g., CLIP-style), or converting visuals into structured representations (tables, JSON).
• Evaluation: measure retrieval quality, grounding, citation correctness, and refusal behavior.

One common mistake is optimizing only the final answer text. That is too late in the pipeline. You want to ask questions like:

• Did retrieval return the right passages?
• Were the passages actually used in the answer?
• Does each cited passage support the claim?
• Did the model refuse when evidence was missing?

These are different problems. If retrieval fails, no prompt will fix it.

Other types of RAG, briefly

What we built here is the most common pattern: vector-based retrieval over chunked text. But it is not the only way to structure retrieval. You'll see terms like:

GraphRAG: Instead of treating documents as independent chunks, GraphRAG models knowledge as entities and relationships (a graph). Retrieval becomes a traversal problem: find relevant nodes, expand neighbors, and assemble context from connected facts. This is useful when your data has structure (people, organizations, events, dependencies) and when relationships matter as much as the text itself. It trades simplicity for control and interpretability.

Agentic RAG: In this setup, retrieval is not a single step. The model can decide how to retrieve: call different tools (vector search, keyword search, APIs), refine the query, or perform multiple hops. What we implemented with function calling is already a minimal version of this idea. A more advanced system lets the model plan retrieval steps, not just execute a single search.

Vector-less RAG: Not all retrieval needs embeddings. In many cases, classical methods like BM25, SQL queries, or structured lookups perform better: exact matches (IDs, error codes, product names), structured data (tables, logs), and small, well-defined corpora. There is also a newer direction often called vectorless RAG, where retrieval is driven more by reasoning and indexing strategies than by embedding similarity.

The interesting question is not “which RAG is the best RAG?” That question leads nowhere useful. The better question is:

What retrieval strategy matches the shape of my knowledge?

Where MCP fits

RAG and MCP (Model Context Protocol) solve different parts of the architecture. RAG is about grounding generation in external knowledge. MCP (see a practical example here) is about exposing tools, resources, and workflows to models through a protocol.

There are a few useful combinations:

• Expose an existing RAG service as an MCP tool, such as rag.search or rag.answer.
• Use MCP as a gateway to internal systems that feed the RAG corpus.
• Expose lower-level RAG operations as tools: embed, upsert, search, rerank, validate.

They operate at different layers of the system, but complement each other.

RAG is about using knowledge. MCP is about accessing capabilities.

Closing thoughts

RAG is a way to move part of the system’s knowledge out of model weights (parametric memory) and into an explicit memory that can be updated, inspected, cited, and governed (non-parametric memory). It is not a magic anti-hallucination button.

It works by retrieving relevant information from non-parametric memory at inference time and conditioning the model on that evidence. It doesn’t make the model “know more.” It changes where knowledge lives and how it is accessed.

The quality of a RAG system depends less on the LLM and more on the unglamorous decisions: corpus design, chunking, metadata, retrieval policy, prompt contract, evaluation, and source handling. When those pieces are done well, RAG stops being “chat with your file” and becomes something much more useful: an interface between language and maintained knowledge.