I Built a RAG System From Scratch. Here’s What the Tutorials Don’t Tell You.

BGE-M3, ChromaDB, FlashrankRerank, DeepSeek R1 and a RAGAS faithfulness score of 0.42 that told me exactly where it fails.

In this blog, I’ll walk you through how I built a full Retrieval-Augmented Generation (RAG) pipeline from scratch using open-source tools and no OpenAI API. Everything runs locally: BGE-M3 for embeddings, ChromaDB for vector storage, FlashrankRerank for re-ranking, and DeepSeek R1 as the LLM. I also ran a proper RAGAS evaluation at the end, which gave me numbers I didn’t expect.

Full code: github.com/tusher16/rag-from-scratch

Why I Built This

I was writing my master’s thesis on LayoutLMv3, a document understanding model, and I needed a way to query the paper and get grounded answers. I could have pointed ChatGPT at the PDF and called it done. But I wanted to understand what was actually happening under the hood: how documents get chunked, why embedding similarity misses things that reranking catches, and what it takes to run this entirely on my own machine.

RAG solves a real problem. LLMs are trained on static data and hallucinate when asked about specific documents. Grounding the generation step in retrieved context dramatically reduces made-up answers [1]. But most tutorials hand-wave over the decisions that actually matter: chunking strategy, retrieval algorithm, reranker config, and how to evaluate the whole thing properly.

Let’s build all four phases.

Architecture

The pipeline has four phases. Ingestion runs once. Retrieval and generation run on every query. Evaluation runs separately to measure how good the system actually is.

graph TD
    subgraph Phase1["Phase 1 — Ingestion (one-time)"]
        A[PDF] --> B[PyPDFLoader]
        B --> C[RecursiveCharacterTextSplitter\n512 chars / 64 overlap]
        C --> D[BGE-M3 Embeddings\nMPS / CPU]
        D --> E[ChromaDB\nPersistent Vector Store]
    end
subgraph Phase2["Phase 2 - Retrieval"]
        F[User Query] --> G[MMR Search\nfetch_k=40 → k=20]
        E --> G
        G --> H[FlashrankRerank\ntop_n=10]
    end
    subgraph Phase3["Phase 3 - Generation"]
        H --> I[DeepSeek R1 8B\nvia LM Studio]
        F --> I
        I --> J[Grounded Answer]
    end
    subgraph Phase4["Phase 4 - Evaluation"]
        K[22 Test Questions] --> L[RAGAS\nFaithfulness + Answer Relevancy]
        J --> L
    end

Diagram created with assistance from ChatGPT.

Figure 1: Four-phase RAG pipeline.

Three design decisions worth explaining up front.

ChromaDB over FAISS : ChromaDB persists to disk automatically and supports metadata filtering. FAISS is faster at scale but needs extra work to persist. For a local research project, ChromaDB is the right tradeoff [2].

BGE-M3 over OpenAI embeddings : BGE-M3 outperforms text-embedding-3-small on multilingual and domain-specific benchmarks and runs entirely on CPU (or MPS on Apple Silicon) [3]. No API cost, no rate limits.

MMR retrieval over simple cosine search : I used Maximal Marginal Relevance instead of plain nearest-neighbor search. MMR fetches a wider candidate pool (40 chunks) then filters for both relevance and diversity before returning the top 20. This matters when your document has repeated concepts plain cosine search returns near-duplicate chunks.

Implementation

Phase 1 : Ingestion

The first question in any RAG system is how to split the document. Too small and you lose context. Too large and the LLM gets flooded with noise.

I settled on 512 characters with 64-character overlap after testing. The recursive splitter tries paragraph breaks first, then sentences, then words it preserves semantic boundaries better than a fixed window.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_documents(documents)
embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-m3",
    model_kwargs={"device": "mps"},   # or "cpu" on non-Apple hardware
    encode_kwargs={"normalize_embeddings": True},
)
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    collection_name="production_rag",
    persist_directory="./vectorstore",
)

One note on imports: newer LangChain versions split packages into langchain_text_splitters, langchain_huggingface, and langchain_chroma. If you copy older tutorials that use langchain.text_splitter or langchain_community.embeddings, the import will fail silently or throw a deprecation error. This cost me about 30 minutes.

Phase 2 — Retrieval with Reranking

This is where the real decisions live.

from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma
from langchain_classic.retrievers import ContextualCompressionRetriever
from langchain_community.document_compressors import FlashrankRerank
from rag_v1 import config

vectorstore = Chroma(
    collection_name=config.COLLECTION_NAME,
    embedding_function=embeddings,
    persist_directory=str(config.VECTORSTORE_DIR),
)
base_retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={
        "k": config.TOP_K,        # return 20
        "fetch_k": config.TOP_K * 2,  # fetch 40 candidates first
        "lambda_mult": 0.7,       # balance relevance vs diversity
    },
)
reranker = FlashrankRerank(top_n=config.RERANK_TOP_N, score_threshold=0.1)
retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=base_retriever,
)

Why RERANK_TOP_N=10? I started at 3. The early eval runs were dropping table chunks the F1 score tables in the paper were garbled by PyPDF, so they scored low in reranking even when they contained the right numbers. Bumping to 10 passed more chunks to the LLM and those answers improved. It was a debugging discovery, not a design choice I made upfront.

Phase 3 — Generation

I used DeepSeek R1 8B running locally via LM Studio. LM Studio serves an OpenAI-compatible endpoint at http://127.0.0.1:1234/v1, so the client is ChatOpenAI with a custom base_url.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough, RunnableLambda

SYSTEM_PROMPT = """You are a precise, grounded assistant.
Answer the question using ONLY the context provided below.
If the answer is not in the context, say: "I don't have enough information."
Never use your own knowledge. Always cite the source.
Context:
{context}"""
def format_context(docs):
    sections = []
    for i, doc in enumerate(docs, 1):
        source = doc.metadata.get("source", "unknown")
        page   = doc.metadata.get("page", "?")
        sections.append(
            f"[Chunk {i} | Source: {source} | Page: {page}]\n{doc.page_content.strip()}"
        )
    return "\n\n---\n\n".join(sections)
prompt = ChatPromptTemplate.from_messages([
    ("system", SYSTEM_PROMPT),
    ("human", "{question}"),
])
chain = (
    {
        "context":  retriever | RunnableLambda(format_context),
        "question": RunnablePassthrough(),
    }
    | prompt
    | llm
    | StrOutputParser()
)

I used LCEL (LangChain Expression Language) instead of RetrievalQA. LCEL gives you explicit control over how context gets formatted before it hits the prompt you can see exactly what the LLM receives, which matters when you're debugging wrong answers.

Phase 4 — Evaluation with RAGAS

Most RAG tutorials skip evaluation entirely. I didn’t want to do that. I built a 22-question test set from the LayoutLMv3 paper covering architecture, datasets, methods, and results, then ran RAGAS [4].

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

# Use a lighter model as judge - DeepSeek R1 was too slow (30-60s per call)
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(
    base_url="http://127.0.0.1:1234/v1",
    api_key="lm-studio",
    model="qwen2.5-3b-instruct-mlx",
    temperature=0,
))
result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy],
    llm=evaluator_llm,
    embeddings=evaluator_embeddings,
    raise_exceptions=False,
    run_config=RunConfig(timeout=120, max_workers=1),
)

22 questions × 2 metrics = 44 RAGAS judge calls. With Qwen2.5 3B each took about 10–20 seconds around 15 minutes total.

What I Ran Into

1. Wrong device name on Apple Silicon. I typed "maps" instead of "mps" in the device argument. The error message pointed at HuggingFace internals and I spent 5 minutes looking in the wrong place before spotting the typo.

2. LangChain import paths had changed. Tutorials from 2023 import from langchain.text_splitter, langchain_community.embeddings, langchain_community.vectorstores. Newer versions moved these to separate packages: langchain_text_splitters, langchain_huggingface, langchain_chroma. Each wrong import either fails silently or throws a deprecation warning that hides the real error.

3. The reranker was dropping table chunks. My first config used RERANK_TOP_N=3. The F1 score tables in the LayoutLMv3 paper were getting garbled by PyPDF the extracted text had broken formatting, so those chunks scored low in reranking. Questions like "What is the best F1 score on FUNSD?" came back wrong because the right chunk never made it past the reranker. I bumped to RERANK_TOP_N=10, passed more chunks to the LLM, and those answers improved.

4. DeepSeek R1 was too slow for RAGAS evaluation. I first tried DeepSeek R1 8B as the RAGAS judge. Each judge call took 30–60 seconds, and some returned 400 Bad Request (context too long). 22 questions × 2 metrics × 60 seconds = over an hour per eval run. I switched to Qwen2.5 3B Instruct as the judge model, which handles the short RAGAS prompts in 10–20 seconds with no context errors.

5. OOM on Metal during evaluation. Running BGE-M3 on MPS for both retrieval and evaluation at the same time caused an out-of-memory crash. The fix was moving the evaluation embeddings to CPU: model_kwargs={"device": "cpu"} in the evaluator config only. The retrieval embeddings stayed on MPS.

Results

RAGAS evaluation on 22 questions from the LayoutLMv3 paper:

Metric Score Faithfulness 0.42 and Answer Relevancy 0.73

What these numbers mean:

Faithfulness at 0.42 means only 42% of the claims in the generated answers are grounded in the retrieved chunks. The other 58% the model is adding from its own training data. This is the problem to fix in v2.

Answer Relevancy at 0.73 means 73% of answers actually address the question asked. Decent for a 3B model.

Why faithfulness is low:

Three reasons I could trace directly from the eval output. First, PyPDF garbles table content the F1 score tables became unreadable chunks that no retriever or reranker could surface cleanly. Second, a 3B LLM doesn’t always follow the “answer only from context” instruction it adds plausible-sounding information even when you explicitly tell it not to. Third, some questions (pre-training dataset composition, Layout Concatenation details) span multiple non-contiguous sections of the paper, and fixed-window chunking splits that context across chunks that don’t get retrieved together.

None of this is surprising in retrospect. These are known limitations of naive RAG [4]. But seeing them show up in your own eval numbers is different from reading about them.

What I’m doing next

For v2, the two changes that should move faithfulness the most are better PDF parsing (switching from PyPDF to unstructured) and hybrid retrieval (BM25 + dense search fused via Reciprocal Rank Fusion). The table questions need keyword matching, not semantic similarity. I'm also planning a Streamlit UI and LangSmith tracing so I can observe every retrieval and generation call.

If you take one thing from this project: add the reranker, and run an actual evaluation. The eval results will show you exactly where your pipeline is failing. Without them you’re just guessing.

Full code: github.com/tusher16/rag-from-scratch

References

[1] P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020. [Online]. Available: https://arxiv.org/abs/2005.11401

[2] Chroma, “Chroma Documentation,” Chroma, 2024. [Online]. Available: https://docs.trychroma.com

[3] BAAI, “BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation,” arXiv, 2024. [Online]. Available: https://arxiv.org/abs/2309.07597

[4] G. Gao et al., “Retrieval-Augmented Generation for Large Language Models: A Survey,” arXiv, 2024. [Online]. Available: https://arxiv.org/abs/2312.10997