Building a Local AI Chatbot using FastAPI, Ollama & TinyLLaMA

“Why just use ChatGPT when you can build your own local chatbot?”

In this post, I’ll walk you through how I built a lightweight AI chatbot that runs entirely on your local machine. It combines the power of Ollama, FastAPI, and a tiny LLM model — TinyLLaMA — to deliver a fast, privacy-friendly conversational AI experience. And yes, it comes with a clean HTML/CSS/JS frontend too!

Motivation

Most chatbot APIs rely on cloud-based models like GPT-3/4, which raise concerns about latency, privacy, and cost. I wanted to explore:

• Running an LLM locally
• Creating a simple API using FastAPI
• Building a minimal UI to make chatting easy
• Keeping chat logs persistent for future use

This project is ideal for developers learning how to integrate LLMs into full-stack applications.

Tech Stack

Component Technology Language Model | TinyLLaMA via Ollama API Backend | FastAPI Frontend | HTML + CSS + JS Model Runtime | Ollama Storage | JSON (for chat logs)

Project Structure

ai-agent-chatbot/
│
├── main.py                # FastAPI app
├── chatlog.json           # Saved conversation history
├── .gitignore             # Ignores .env and logs
├── frontend/
│   ├── index.html         # Chat UI
│   ├── script.js          # Handles API calls
│   └── styles.css         # UI styling (optional)

Step-by-Step: How I Built It

1. Start a Local LLM with Ollama

Ollama is a local model runner — super easy to install.

ollama run tinyllama

This will keep the model running on [http://localhost:11434.](http://localhost:11434.)

2. FastAPI Backend

Here’s what the backend handles:

• Receives chat messages via /api/chat
• Sends them to the Ollama API
• Logs the conversation into chatlog.json
• Supports chat reset via /api/reset
• Serves the static frontend

@app.post("/api/chat")
async def chat(request: Request):
    body = await request.json()
    message = body.get("message")

    conversation_history.append({"role": "user", "content": message})

    response = requests.post("http://localhost:11434/api/chat", json={
        "model": "tinyllama",
        "messages": conversation_history,
        "stream": False
    })

    if response.status_code == 200:
            reply = response.json()["message"]["content"]
            conversation_history.append({"role": "assistant", "content": reply})
            save_history_to_file()
            return {"response": reply}

        return {"error": "Model failed to respond"}

3. The Frontend UI

The frontend is simple but functional:

const response = await fetch("http://localhost:8000/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ message: message })
});

It displays both user and assistant messages and handles chat resets.

Secret Scanning Lesson

While pushing to GitHub, I ran into a roadblock: GitHub rejected my push because my .env file (with a HuggingFace token) was committed. GitHub's secret scanning caught it.

To fix it:

1. I removed the .env file.
2. Added it to .gitignore.
3. Used git filter-repo to scrub it from history

git filter-repo --path .env --invert-paths --force

4. Re-pushed the repository.

Key Takeaways

• You don’t need cloud APIs to build an LLM-based chatbot.
• Ollama + FastAPI is a simple but powerful combo.
• Frontend + backend integration teaches full-stack skills.
• GitHub secret scanning is a blessing in disguise!

Demo + Source Code

🔗 GitHub: github.com/shrehs/ai-agent-chatbot

Feel free to fork, clone, or customize it with your own model and frontend!

Final Thoughts

This project made me realize how accessible LLMs are becoming. With tools like Ollama and small models like TinyLLaMA, anyone can build private, local-first AI tools.

If you’re interested in AI, full-stack dev, or open-source — this is a great weekend project.

If you liked this, follow me on GitHub or connect with me for future updates.

Let me know if you’d like the Markdown version of this or want help publishing it on Medium!