Building a Prototype RAG Pipeline from Scratch using LangChain, HuggingFace & ChromaDB

Retrieval-Augmented Generation (RAG) has become one of the core architectures behind modern AI applications. In this article, I’ll walk through the implementation of a prototype RAG pipeline and explain the role of each component, the design decisions behind it, and some of the alternatives that can be considered depending on the use case.

This implementation is intentionally a prototype. The goal was to build a clean end-to-end retrieval pipeline first before moving on to production-oriented improvements like hybrid retrieval, reranking, metadata filtering, evaluation, and monitoring.

Rather than only sharing the final code, I’ll also explain why each component exists, what role it plays in the pipeline, and some of the alternatives that can be considered depending on the use case.

Architecture

The current pipeline looks like this:

PDF Documents
      │
      ▼
PyMuPDFLoader
      │
      ▼
RecursiveCharacterTextSplitter
      │
      ▼
Embedding Model
      │
      ▼
ChromaDB
      │
      ▼
Semantic Retrieval

Let’s go through each stage.

1. Loading Documents

The first step is getting the data into the pipeline.

For this implementation, I’m using PDF documents and loading them recursively so the pipeline automatically picks up every PDF inside the data directory.

pdf_files = sorted(pdf_dir.glob("**/*.pdf"))

For text extraction, I chose PyMuPDFLoader.

loader = PyMuPDFLoader(str(pdf_file))
documents.extend(loader.load())

PyPDFLoader would also work, but I found PyMuPDFLoader to be a better fit for this project because it's generally faster and performs well with the documents I tested.

2. Chunking

Once the documents are loaded, they need to be divided into smaller pieces.

Instead of embedding an entire document, I split it into overlapping chunks using:

RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)

The overlap helps preserve context between neighbouring chunks, while recursive splitting tries to split on natural boundaries such as paragraphs and sentences before falling back to individual words or characters.

Other chunking strategies are also available, including fixed-size chunking and semantic chunking. Which one works best depends on the data and the retrieval requirements.

3. Creating Embeddings

Once the documents have been split into chunks, the next step is converting each chunk into a vector representation.

This is where the embedding model comes into the picture.

For this prototype, I initially experimented with BGE models, but for local development I switched to **sentence-transformers/all-MiniLM-L6-v2** since it's lightweight and runs comfortably on my machine.

from langchain_huggingface import HuggingFaceEmbeddings

embedding_model = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2",
    model_kwargs={"device": "mps"},
    encode_kwargs={"normalize_embeddings": True},
)

One thing I found interesting while building this is that the embedding layer is completely independent of the rest of the pipeline.

The pipeline doesn’t really care which embedding model you’re using. As long as the model can generate embeddings, it can be plugged into the same architecture.

For example, instead of using a local HuggingFace model, the embedding layer could also use:

Local Models

BAAI BGE

Nomic Embed

Ollama Embeddings

Sentence Transformers

API-based Models

• OpenAI text-embedding-3-small
• OpenAI text-embedding-3-large
• Cohere Embed
• Voyage AI

Choosing between a local model and an API usually comes down to factors like cost, latency, infrastructure, and deployment requirements.

For this prototype, I wanted everything to run locally.

What exactly is an embedding?

An embedding is simply a numerical representation of text.

For example,

"What is Retrieval-Augmented Generation?"

might become something like:

[-0.18, 0.42, -0.07, 0.93, ...]

These numbers don’t mean much to us, but they allow the system to compare the semantic meaning of different pieces of text.

That’s what makes semantic search possible.

4. Storing Embeddings in ChromaDB

Once the embeddings are generated, they need to be stored somewhere.

For this project, I used ChromaDB as the vector database.

from langchain_chroma import Chroma

vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embedding_model,
    persist_directory="../chroma_db"
)

One thing that became much clearer while implementing this pipeline is the difference between the embedding model and the vector database.

The embedding model is responsible for converting text into vectors.

The vector database doesn’t understand language or generate embeddings, its responsibility is to efficiently store vectors and retrieve the ones that are most similar to a query vector.

There are several vector databases available today, including:

• ChromaDB
• Pinecone
• Quadrant
• Milvus
• Weaviate

For this prototype, ChromaDB was a good choice because it’s lightweight, runs locally, and integrates nicely with LangChain.

5. Semantic Retrieval

After indexing the documents, the retrieval pipeline is ready.

Suppose the user asks:

"What is marks?"

Searching doesn’t happen by comparing text directly.

The first step is converting the user’s query into an embedding using the same embedding model that was used while indexing the documents.

query = "What is marks"

results = vector_store.similarity_search(
    query,
    k=3
)

Internally, the flow looks something like this:

User Query
      │
      ▼
Embedding Model
      │
      ▼
Query Embedding
      │
      ▼
ChromaDB
      │
      ▼
Most Similar Chunks

Because both the documents and the query are represented in the same vector space, the system can retrieve relevant chunks even when the wording isn’t exactly the same.

That’s the key difference between traditional keyword search and semantic search.

What’s Next?

At this stage, the retrieval pipeline is complete.

The next step is connecting an LLM so that the retrieved chunks can be used as context while generating the final answer.

The high-level flow would then become:

PDF Documents
      │
      ▼
Document Loader
      │
      ▼
Chunking
      │
      ▼
Embedding Model
      │
      ▼
ChromaDB
      │
      ▼
Semantic Retrieval
      │
      ▼
LLM
      │
      ▼
Generated Response

Once the basic pipeline is working, there are several improvements that can be made before calling it production-ready.

Some of the areas I plan to explore next are:

• Hybrid Retrieval (Dense + BM25)
• Better embedding models
• Metadata filtering
• Cross-encoder reranking
• Prompt engineering
• Evaluation
• Monitoring