I Built a Local Voice AI Agent From Scratch. Here’s What Actually Happened.

Whisper, Ollama, FastAPI, a chat UI that looks like Claude, and more Windows encoding bugs than I care to admit.

I gave myself a weekend. The goal was a voice agent that listens to what you say, figures out what you want, and does it. Write a file, generate some code, summarize a document. All local. No API keys, no monthly bill.

Not because any single part is hard. The hard part is that every layer breaks in a different way, and the bugs don’t happen in isolation. By Saturday night I had a working demo. By Sunday night I had something I’d actually use. This is what I built, what broke, and what I learned.

What it does

The finished system handles a few things:

• Voice input via microphone or uploaded audio file

• Text input if you just want to type

• File attachments: PDFs, images, code files, CSVs

• Four intents: write code, create a file, summarize text, general chat

• Everything saved to output/ and nowhere else

The UI looks like a chat interface. Dark sidebar, conversation in the center, input bar at the bottom with a plus button for files, a textarea, a mic button, and send. Pretty much what you’d expect if you’ve used ChatGPT or Claude.

The stack

Four layers, each one independent enough that you can swap it out:

Speech-to-text: faster-whisper

I went with faster-whisper instead of the original Whisper library because it runs on CPU without complaint. The tiny model is 75MB and transcribes a 5-second clip in 3 or 4 seconds on a regular laptop. Good enough for a demo.

I also built a Groq fallback. If you set GROQ_API_KEY in your environment, the app routes STT through Groq’s hosted Whisper API instead. Free tier, near-instant, same quality. Useful when you’re presenting something and don’t want to sit through 4 seconds of silence after every recording.

Intent classification: Ollama + llama3

This is where most of the interesting work is. The transcribed text goes to a local Ollama instance with a prompt that asks for JSON back. Intent, confidence score, suggested filename, programming language if relevant.

Getting a small local model to return clean JSON consistently is not as easy as it sounds. My first few prompts got JSON wrapped in markdown fences, or JSON with a paragraph of explanation before it, or JSON with extra fields I didn’t ask for. The thing that finally worked was being extremely explicit:

Respond ONLY with valid JSON, no markdown, no explanation.

That last sentence does most of the work. Models seem to treat it as a hard constraint rather than a suggestion.

I also wrote a keyword-based rule classifier as a fallback. If Ollama is offline or the model returns garbage, the rules kick in. It’s not smart but it handles the common cases and doesn’t crash.

Tool execution: sandboxed to output/

Once the intent is classified, the matching handler runs. All four handlers (write code, create file, summarize, chat) use the same _safe_path() function to resolve where to write. It strips directory components from filenames and forces everything into output/. You can’t path-traverse your way out of it.

The LLM generates the actual content via a second Ollama call. For code, there’s a regex pass afterwards to strip markdown fences in case the model added them despite being told not to.

The frontend: one HTML file

No React. No build step. One HTML file with CSS and JS inline, served by FastAPI’s static files. I rewrote the UI three times before landing on something I liked.

The input bar was the trickiest part to get right. I wanted everything on one row: file attach, text input, mic, send. The challenge is that the textarea needs to auto-resize as you type, the mic button needs to animate while recording, and the whole thing needs to not feel cramped. A few iterations of flexbox later, it works.

The bugs that cost me the most time

Windows encoding crash, day one

First run, first error: UnicodeDecodeError: ‘charmap’ codec can’t decode byte 0x8f. FastAPI was reading the HTML file using Windows CP-1252 by default. The HTML file had some Unicode characters in it. The entire server crashed on every request.

The fix was one extra argument:

html_path.read_text(encoding='utf-8')

I spent about an hour on this before I figured out what was happening.

Lesson: always specify encoding explicitly on Windows.

Environment variables not sticking in Git Bash

I kept setting set OLLAMA_MODEL=llama3 in Git Bash and wondering why the server was still trying to use llama3.2. Turns out Git Bash uses Unix-style syntax. The Windows set command does nothing there.

# This does nothing in Git Bash:
set OLLAMA_MODEL=llama3
# This works:
export OLLAMA_MODEL=llama3

Simple once you know it. Annoying to figure out.

History sidebar stacking conversations

Clicking a history item was appending the result to whatever was already on screen instead of replacing it. So if you clicked two different history items, you’d see both conversations merged together on the same page.

The fix was to clear the message list before rendering:

d.onclick = () => {
document.getElementById('msgs').innerHTML = '';
showMsgs();
addUserMsg(data.transcription, []);
renderResult(data);
};

One line. Cost me twenty minutes of confused staring at the screen.

Voice recording showing [Voice recording] instead of what I said

The user message bubble was being created before the server responded, so it always showed a placeholder. I needed to create the bubble early (so the UI feels responsive) but update the text once the transcript came back.

// Create bubble immediately with placeholder
el.id = 'voice-transcript';
el.textContent = 'Transcribing…';
// Update once server responds
document.getElementById('voice-transcript')
.textContent = response.transcription;

PDF and file handling

Adding file support meant writing a file_reader.py module that routes uploads by type. PDFs go through pdfplumber for page-by-page text extraction. Images get base64-encoded and sent to llava, a local vision model running through Ollama. Text and code files just get decoded as UTF-8.

The extracted content gets injected into the LLM prompt alongside whatever the user typed. So you can attach a PDF and type ‘summarize the key points’ and the full text lands in the context window.

The main limitation is context length. A 50-page PDF is a lot of tokens for a 4B model. It works for shorter documents. For longer ones, you’d want to chunk it or summarize sections separately before combining.

Notes on the models

Whisper: tiny is fine for demos

The tiny model handles clear speech in a quiet room well. It struggles with heavy accents and background noise. base is noticeably better and still fast enough on CPU. small is close to human-level but takes 10–15 seconds per clip on CPU, which kills the interactive feel. Stick with tiny or base unless you have a GPU.

llama3 vs phi3 for intent classification

llama3 (4.7GB) returned clean JSON more reliably than phi3 (2.3GB). phi3 was faster but occasionally added extra fields or misclassified edge cases. If RAM is tight, phi3 works fine for straightforward commands. Just be more explicit in your prompt and add a few examples of the expected output format.

llava for images

llava is optional but useful. Send it a base64 image with a prompt and it describes what it sees. For screenshots with text it does a decent transcription job. If you don’t have it installed the agent tells you and moves on.

What I’d change

Streaming

Right now the typing indicator spins until the full response arrives, then everything appears at once. Ollama’s API supports streaming. Wiring it up would make the whole thing feel much faster even when the model is slow. It’s probably the highest-impact change I could make.

Conversation memory

Every request is stateless. The model has no memory of the last message. Adding a rolling message history and passing it with each request would enable follow-up questions. Ollama’s /api/chat endpoint takes a messages array, so the plumbing is already there.

Confirm before writing files

Before creating anything in output/, the agent could show a preview and ask for confirmation. It would slow things down slightly but make it much safer to demo in front of people who aren’t sure what the agent is about to do.

Closing

Local models have gotten genuinely good. llama3 on a CPU handles structured tasks well enough that I’d reach for it in real projects. The latency is noticeable but not painful.

Most of the time I spent on this project wasn’t on the AI parts. It was on encoding bugs, env variable syntax, UI layout edge cases, and one-line fixes that took an hour to find. That’s just software.

If I were to hand this to someone and say ‘extend it’, I’d say: add streaming first. Everything else is additive. Streaming changes the feeling of the whole thing in a way that no accuracy improvement does.

Code is on GitHub. Runs at localhost:8000. Nothing leaves your machine unless you set a Groq API key.

Stack: Python + FastAPI, faster-whisper, Ollama (llama3, llava), pdfplumber, vanilla JS. Built for the Mem0 Generative AI Developer Intern assignment.