Add the RAG layer

In continuation to the previous blog, an expanded deep dive on Step 4: Add a RAG Layer (Optional — but gives AI memory) for the air-gapped Ollama AI setup based on requests by the readers…

Retrieval-Augmented Generation (RAG) combines the power of large language models (LLMs) with an external knowledge base or document store. Instead of relying solely on the model’s internal weights, it retrieves relevant contextual information from your private, offline database and feeds that to the LLM to generate better, more accurate, and context-aware answers.

This is especially critical in air-gapped environments where:

• Your AI doesn’t have access to the internet or external knowledge bases.
• You want it to “remember” your proprietary, confidential, or dynamic knowledge.
• You need personalized or domain-specific responses.

How to Add RAG in Your Air-Gapped Setup

1. Choose Your Local Vector Database

Select an open-source, container-friendly, offline-capable vector store to embed and index your documents:

• Chroma — Lightweight and easy to run locally.
• Milvus — Scalable, feature-rich, supports high-dimensional data.
• Weaviate — Also supports offline setups with proper configuration.

Make sure the vector store supports offline operation and can run inside the air-gapped network.

2. Embed Your Secret Docs Locally

• Preprocessing: Clean, tokenize, and chunk your documents (PDFs, text files, reports).
• Generate embeddings: Use an embedding model compatible with Ollama (or open-source ones you can run offline).
• Index: Insert these embeddings into your vector database on the air-gapped server.

This creates a local searchable knowledge base of your sensitive data — completely offline.

3. Run a Retriever API Over the Secure Network

Deploy a REST or gRPC API that queries your vector store and returns relevant document snippets or contexts based on user queries.

Example setup:

The API runs inside the same secure podman network (ollama-secure-net).

Ollama queries this API whenever it receives a prompt requiring external knowledge.

The retriever fetches matching documents from the vector DB.

4. Chain Inputs into Ollama

Create a pipeline that:

• Takes user input.
• Sends a query to the retriever API.
• Combines the retrieved context with the original prompt.
• Feeds this enriched prompt to the Ollama LLM container for final generation.

This pipeline can be implemented in Python using the Ollama SDK, or even shell scripts calling REST endpoints.

Benefits of This Architecture

• Memory & Context: Your AI now “remembers” the data you feed it, beyond the static model parameters.
• Customization: Embed proprietary manuals, client data, compliance rules — all offline.
• Improved Accuracy: Contextually relevant answers that adhere to your unique knowledge base.
• Full Data Control: Your secrets never leave your secure environment.

Practical Tips

• Keep your embeddings and vector DB updated regularly through secure offline transfers.
• Use chunking and summarization techniques to fit retrieved content within Ollama’s token limits.
• Secure the retriever API with mutual TLS and authentication inside your internal network.
• Monitor latency and optimize vector DB parameters for smooth querying.

A sample Python pipeline to integrate Ollama with a local vector database retriever API in an air-gapped environment, plus a simple Podman container orchestration snippet to run everything inside your secure network.

This example assumes:

• You have a local retriever API running at http://localhost:5000/retrieve that accepts POST requests with JSON { “query”: “your question” } and returns relevant context.
• Ollama LLM is running locally and accessible via CLI commands.
• Python 3.8+ installed with requests package.

import requests
import subprocess
def query_retriever(query: str) -> str:
    """Send query to local vector DB retriever API and get context."""
    url = "http://localhost:5000/retrieve"
    payload = {"query": query}
    try:
        response = requests.post(url, json=payload, timeout=10)
        response.raise_for_status()
        data = response.json()
        # Assume API returns {"context": "... relevant text ..."}
        return data.get("context", "")
    except Exception as e:
        print(f"Retriever API error: {e}")
        return ""
def ask_ollama(prompt: str) -> str:
    """Call Ollama CLI locally with combined prompt."""
    try:
        # Construct prompt; adjust for your Ollama model and CLI options
        result = subprocess.run(
            ["ollama", "chat", "llama2-7b"],
            input=prompt.encode(),
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            timeout=30,
        )
        if result.returncode == 0:
            return result.stdout.decode()
        else:
            print(f"Ollama error: {result.stderr.decode()}")
            return ""
    except Exception as e:
        print(f"Error running Ollama: {e}")
        return ""
def main():
    user_question = input("Enter your question: ")
    # Step 1: Retrieve context
    context = query_retriever(user_question)
# Step 2: Combine user question + retrieved context for prompt
    combined_prompt = (
        f"Context:\n{context}\n\n"
        f"Answer the following question based on the context above:\n{user_question}"
    )
# Step 3: Ask Ollama with enriched prompt
    answer = ask_ollama(combined_prompt)
    print("\nAI Answer:\n", answer)
if __name__ == "__main__":
    main()

Podman Compose: Run Ollama + Retriever API on Secure Network

Save this as docker-compose.yml (compatible with Podman Compose):

version: "3.8"
networks:
  ollama-secure-net:
    internal: true
volumes:
  ollama-data:
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama-secure
    volumes:
      - ollama-data:/root/.ollama
    ports:
      - "11434:11434"
    networks:
      - ollama-secure-net
    restart: unless-stopped
retriever:
    image: your-retriever-image:latest
    container_name: retriever-api
    ports:
      - "5000:5000"
    networks:
      - ollama-secure-net
    restart: unless-stopped

How to Use:

1. Build and run your retriever API as a container image (your-retriever-image) that exposes /retrieve endpoint.
2. Use the above compose file with Podman Compose:

podman-compose up -d

Run your Python pipeline on the same air-gapped machine. It will query the retriever API and send enriched prompts to Ollama locally.

Happy Building, Continuous Learning and Stay Secure! 🔐🤖