I Built a Fully Offline AI Agent That Answers Questions From PDF, Images, and Audio — No Cloud Required

Building a complete RAG system with OCR, Whisper, CLIP, and open-source language models on a mid-tier laptop

The Hook: Why Build Local When Cloud is King?

Picture this: You're on a 12-hour flight with sensitive company documents, need to analyze audio recordings from a confidential meeting, and have images of handwritten notes that contain crucial insights. Your usual go-to? ChatGPT or Claude. But there's no internet, and even if there was, uploading proprietary data to external APIs isn't an option.

This is exactly why I built my own fully offline AI agent — and why you should consider doing the same.

Cloud LLMs are incredible, but they're not always the answer. Sometimes you need:

• Complete data privacy (no external API calls)
• Zero internet dependency (planes, remote locations, secure environments)
• Cost control (no $3,000 surprise bills for heavy usage)
• Full system ownership (understand every component)

The Challenge: Building a Multimodal AI Assistant

I wanted to create an AI agent that could seamlessly handle three types of input:

1. PDF documents (reports, papers, invoices)
2. Images with text (scanned documents, handwritten notes, charts)
3. Audio files (lectures, meetings, interviews)

All while running entirely on my laptop — no cloud dependencies.

The Architecture: A Fully Local Tech Stack

After evaluating various options, here's the stack I assembled:

• LLM: Mistral 7B via Ollama (local inference)
• Embeddings: sentence-transformers (local vector generation)
• OCR: Tesseract (image-to-text conversion)
• Speech-to-Text: OpenAI Whisper (local audio transcription)
• Document Parsing: PyMuPDF (PDF text extraction)
• Vector Storage: FAISS (similarity search)
• Orchestration: LangChain (RAG pipeline)
• Interface: Streamlit (web UI) + Terminal

No APIs. No subscriptions. Just pure local compute.

Building the System: A Step-by-Step Journey

Step 1: PDF Text Extraction with PyMuPDF

Most valuable information lives in PDF documents. The first challenge was making them machine-readable:

import fitz  # PyMuPDF

def extract_text_from_pdf(pdf_path):
    doc = fitz.open(pdf_path)
    full_text = ""
    for page_num, page in enumerate(doc):
        text = page.get_text()
        full_text += f"\n[Page {page_num + 1}]\n{text}"
    doc.close()
    return full_text

This simple function became the foundation for processing research papers, financial reports, and technical documentation.

Step 2: OCR for Scanned Images

Many PDFs contain scanned images with text that's invisible to standard parsers. Tesseract OCR solved this:

import pytesseract
from PIL import Image

def extract_text_from_image(image_path):
    try:
        image = Image.open(image_path)
        extracted_text = pytesseract.image_to_string(image)
        return extracted_text.strip()
    except Exception as e:
        return f"Error processing image: {str(e)}"

Suddenly, handwritten notes and scanned documents became searchable knowledge.

Step 3: Audio Transcription with Whisper

I had hours of recorded lectures and meeting audio. Whisper handled them brilliantly:

import whisper

# Load model once (cached locally)
whisper_model = whisper.load_model("base")

def transcribe_audio(audio_path):
    result = whisper_model.transcribe(audio_path)
    return result['text']

The transcription quality was remarkable, even with background noise and multiple speakers.

Step 4: Intelligent Text Chunking

Raw text needed to be broken into manageable, searchable chunks while preserving context:

def chunk_text_with_metadata(text, source_info, chunk_size=1000, overlap=200):
    chunks = []
    for i in range(0, len(text), chunk_size - overlap):
        chunk = text[i:i + chunk_size]
        chunk_metadata = {
            'content': chunk,
            'source': source_info['filename'],
            'type': source_info['type'],  # 'pdf', 'image', 'audio'
            'chunk_index': len(chunks),
            'start_pos': i
        }
        chunks.append(chunk_metadata)
    return chunks

This metadata proved crucial for providing context in responses.

Step 5: Local Embedding Generation

Converting text to vectors using sentence-transformers:

from sentence_transformers import SentenceTransformer

# Load embedding model (runs locally)
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

def generate_embeddings(text_chunks):
    texts = [chunk['content'] for chunk in text_chunks]
    embeddings = embedding_model.encode(texts)
    return embeddings

No OpenAI embedding API required — everything computed locally.

Step 6: FAISS Vector Storage

For fast similarity search across thousands of chunks:

import faiss
import numpy as np
import pickle

def build_vector_index(embeddings, metadata):
    dimension = embeddings.shape[1]
    index = faiss.IndexFlatL2(dimension)
    index.add(embeddings.astype('float32'))

    # Save index and metadata
    faiss.write_index(index, "local_index.faiss")
    with open("metadata.pkl", "wb") as f:
        pickle.dump(metadata, f)

    return index

Step 7: The RAG Pipeline with LangChain

Connecting all pieces into a question-answering system:

from langchain.chains import RetrievalQA
from langchain.vectorstores import FAISS
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.llms import Ollama

def setup_qa_chain():
    # Initialize local components
    embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
    vectorstore = FAISS.load_local("./vector_store", embeddings)
    llm = Ollama(model="mistral:7b")

    # Create QA chain
    qa_chain = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=vectorstore.as_retriever(search_kwargs={"k": 5})
    )
    return qa_chain

# Usage
qa_system = setup_qa_chain()
response = qa_system.run("What were the key findings from the Q4 earnings call?")

Step 8: Visual Search with CLIP

Adding semantic image search capabilities:

from transformers import CLIPProcessor, CLIPModel
import torch

clip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
clip_processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

def search_images_by_text(query, image_paths):
    # Process query
    text_inputs = clip_processor(text=query, return_tensors="pt")
    text_features = clip_model.get_text_features(**text_inputs)

    best_matches = []
    for img_path in image_paths:
        image = Image.open(img_path)
        image_inputs = clip_processor(images=image, return_tensors="pt")
        image_features = clip_model.get_image_features(**image_inputs)

        # Calculate similarity
        similarity = torch.cosine_similarity(text_features, image_features)
        best_matches.append((img_path, similarity.item()))

    return sorted(best_matches, key=lambda x: x, reverse=True)

Now I could search for "revenue chart" and find relevant diagrams across hundreds of images.

Creating a Streamlit Interface

To make the system more user-friendly, I built a Streamlit application:

import streamlit as st
import os
from pathlib import Path

def main():
    st.set_page_config(
        page_title="Offline AI Agent",
        page_icon="🤖",
        layout="wide"
    )

    st.title("🤖 Offline Multimodal AI Agent")
    st.markdown("*Ask questions about your PDFs, images, and audio files - completely offline*")

    # Sidebar for file uploads
    with st.sidebar:
        st.header("📁 Upload Files")

        uploaded_files = st.file_uploader(
            "Choose files",
            type=['pdf', 'png', 'jpg', 'jpeg', 'wav', 'mp3', 'mp4'],
            accept_multiple_files=True
        )

        if st.button("Process Files"):
            if uploaded_files:
                process_uploaded_files(uploaded_files)
                st.success("Files processed successfully!")

    # Main chat interface
    st.header("💬 Ask Your AI Agent")

    # Initialize chat history
    if "messages" not in st.session_state:
        st.session_state.messages = []

    # Display chat messages
    for message in st.session_state.messages:
        with st.chat_message(message["role"]):
            st.markdown(message["content"])

    # Chat input
    if prompt := st.chat_input("Ask a question about your files..."):
        # Add user message
        st.session_state.messages.append({"role": "user", "content": prompt})
        with st.chat_message("user"):
            st.markdown(prompt)

        # Generate response
        with st.chat_message("assistant"):
            with st.spinner("Thinking..."):
                response = get_ai_response(prompt)
                st.markdown(response)
                st.session_state.messages.append({"role": "assistant", "content": response})

def process_uploaded_files(files):
    """Process uploaded files and update the vector store"""
    progress_bar = st.progress(0)

    for i, file in enumerate(files):
        # Save uploaded file temporarily
        temp_path = f"temp_{file.name}"
        with open(temp_path, "wb") as f:
            f.write(file.read())

        # Process based on file type
        if file.type == "application/pdf":
            text = extract_text_from_pdf(temp_path)
            process_and_store_text(text, file.name, "pdf")
        elif file.type.startswith("image/"):
            text = extract_text_from_image(temp_path)
            process_and_store_text(text, file.name, "image")
        elif file.type.startswith("audio/"):
            text = transcribe_audio(temp_path)
            process_and_store_text(text, file.name, "audio")

        # Clean up
        os.remove(temp_path)
        progress_bar.progress((i + 1) / len(files))

def get_ai_response(query):
    """Get response from the local AI agent"""
    try:
        qa_chain = setup_qa_chain()
        response = qa_chain.run(query)
        return response
    except Exception as e:
        return f"Sorry, I encountered an error: {str(e)}"

if __name__ == "__main__":
    main()

Advanced Features

Voice Input Integration

import speech_recognition as sr
import streamlit as st

def add_voice_input():
    if st.button("🎤 Record Voice Question"):
        recognizer = sr.Recognizer()
        with sr.Microphone() as source:
            st.info("Listening... Speak your question now!")
            audio = recognizer.listen(source, timeout=5)

        try:
            query = recognizer.recognize_google(audio)
            st.success(f"You said: {query}")
            return query
        except sr.UnknownValueError:
            st.error("Could not understand the audio")
            return None

Persistent Chat History

import json
from datetime import datetime

def save_chat_history(messages):
    timestamp = datetime.now().isoformat()
    history_entry = {
        "timestamp": timestamp,
        "messages": messages
    }

    # Load existing history
    try:
        with open("chat_history.json", "r") as f:
            history = json.load(f)
    except FileNotFoundError:
        history = []

    history.append(history_entry)

    # Save updated history
    with open("chat_history.json", "w") as f:
        json.dump(history, f, indent=2)

A message from our Founder

Hey, Sunil here. I wanted to take a moment to thank you for reading until the end and for being a part of this community.

Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community. ❤️

If you want to show some love, please take a moment to follow me on LinkedIn, TikTok, **Instagram. You can also subscribe to our [weekly newsletter](https://newsletter.plainenglish.io/)**.

And before you go, don’t forget to clap and follow the writer️!