Building a Simple RAG Pipeline Using Local LLMs (Ollama + ChromaDB)

In the era of AI-powered applications, Retrieval-Augmented Generation (RAG) has emerged as a powerful pattern to build intelligent systems that can answer questions using your own data.

In this article, we’ll walk through a simple yet effective RAG pipeline using:

• Ollama (for local LLM inference)
• ChromaDB (for semantic search)
• LangChain
• Python

This setup is ideal for enterprise use cases where data privacy matters, since everything runs locally.

What is RAG?

RAG (Retrieval-Augmented Generation) is a technique where:

1. You store your data as embeddings
2. Retrieve relevant chunks based on a query
3. Pass them to an LLM to generate accurate answers

This avoids hallucination and makes AI responses grounded in your data.

Architecture Overview

Tech Stack

• LLM: Ollama (Llama3)
• Vector DB: ChromaDB
• Orchestration: LangChain
• Language: Python

Setup Instructions

1. Install Ollama

https://ollama.com/download

2. Pull the model

ollama pull llama3

Github: AnilKr007/rag-ollama-chroma: Built a proof-of-concept to explore how local LLMs can be used for retrieval-augmented generation (RAG) using Ollama and ChromaDB.

Install dependencies

py -m pip install -r requirements.txt

Step 1: Document Ingestion

The ingestion pipeline reads .txt files, splits them, and stores embeddings.

Key responsibilities:

• Load documents
• Split into chunks
• Generate embeddings
• Store in vector DB

Core logic:

documents = load_documents()
split_docs = splitter.split_documents(documents)
vectorstore = Chroma.from_documents(
    documents=split_docs,
    embedding=embedding,
    persist_directory=CHROMA_DB_DIR

Highlights:

• Uses RecursiveCharacterTextSplitter
• Adds metadata (source file)
• Stores embeddings locally

Run it:

py src/ingest.py

Step 2: Query Pipeline

Once data is indexed, we can query it using semantic search.

Flow:

1. Load vector DB
2. Retrieve top-K similar chunks
3. Send context to LLM
4. Generate answer

RAG Chain:

rag_chain = (
    {
        "context": retriever | format_docs,
        "question": RunnablePassthrough()
    }
    | prompt
    | llm
    | StrOutputParser()
)

Prompt Design

Answer the question based only on the context below.

This ensures:

• No hallucination
• Context-aware answers

Run query:

py src/query.py

Full Pipeline Flow

1. Load documents
2. Split into chunks
3. Generate embeddings
4. Store in ChromaDB
5. User asks question
6. Retrieve relevant chunks
7. Send context to LLM
8. Generate response

Features

• Load and process documents
• Smart chunking
• Local embeddings via Ollama
• Semantic search using ChromaDB
• Context-aware LLM responses
• Fully local (privacy-friendly)

Why This Matters for Enterprise

This architecture is powerful because:

Data Privacy

No external API calls — everything runs locally.

Cost Efficient

No token-based billing.

Extensible

Can scale to:

• Multiple data sources
• APIs
• Databases

Foundation for Advanced Systems

This is the base for:

• Chatbots
• Knowledge assistants
• Document intelligence systems

Limitations

• Local models are slower than cloud LLMs
• Limited context window
• Requires optimization for large datasets

Future Enhancements

• Add support for PDFs & APIs
• Introduce hybrid search (keyword + vector)
• Add UI (React-based chatbot)
• Integrate better LLMs (cloud or fine-tuned)

Final Thoughts

This project demonstrates how you can build a complete RAG system locally using modern tools.

It’s a great starting point for building:

• Enterprise AI assistants
• Internal knowledge systems
• Secure AI applications