Build Your Own Local ChatGPT Using Ollama + Chainlit with Docker Compose 🚀

Large Language Models are becoming part of everyday development workflows. But most developers rely on cloud APIs like OpenAI or Google, which require API keys, internet connectivity, and sometimes expensive usage costs.

What if you could run a ChatGPT-like assistant completely on your local machine?

In this guide, we will build a simple local AI chat application using:

• Ollama — to run LLM models locally
• Chainlit — to create a chat interface
• Docker Compose — to orchestrate containers

By the end, you will have your own private ChatGPT running locally.

Generated using chatgpt

🚨 Hiring Tech Talent (Remote and Onsite) 💰 $3K–$10K/Month

**👉 Apply in 60 seconds**

Why Run AI Locally?

Running models locally has several advantages:

✅ Privacy — Your prompts stay on your machine ✅ No API costs — No pay-per-token billing ✅ Offline usage — Works without internet after model download ✅ Developer control — Easily integrate with your apps

With modern open-source models like:

• Gemma
• Mistral
• Llama

you can run surprisingly powerful AI directly on your laptop.

Architecture Overview

Our setup is simple and lightweight. Find whole code here: AI_Agents/local_chatgpt at main · artiguptaa/AI_Agents

Browser
   │
   ▼
Chainlit (Chat UI)
   │
   ▼
Ollama API
   │
   ▼
Local LLM Model

Two containers will work together:

Ollama Container

• Runs AI models
• Exposes API on port 11434

Chainlit Container

• Provides a web chat interface
• Sends prompts to Ollama

Project Structure

Create a simple project directory:

local_chatgpt/
├── docker-compose.yml
├── Dockerfile
├── requirements.txt
└── app.py

Step 1: Docker Compose Configuration

Create docker-compose.yml.

version: "3.9"
services:

  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Uncomment below if you have NVIDIA GPU
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 10s
      retries: 10
      start_period: 30s

  chainlit:
    image: python:3.11
    container_name: chainlit_app
    working_dir: /app
    volumes:
      - .:/app
    ports:
      - "8000:8000"
    depends_on:
      ollama:
        condition: service_healthy
    environment:
      - OLLAMA_HOST=http://ollama:11434
    command: >
      bash -c "
      pip install pydantic==2.10.1 chainlit ollama &&
      python -c 'import ollama; ollama.pull(\"gemma3:4b\")' &&
      chainlit run app.py -w --host 0.0.0.0 --port 8000
      "

volumes:
  ollama:

This creates two services:

• Ollama container for the LLM
• Chainlit container for the UI

Step 2: Create the Chainlit App

Create app.py.

import chainlit as cl
import asyncio
import ollama

@cl.on_chat_start
async def start_chat():
    cl.user_session.set(
        "interaction",
        [
            {
                "role": "system",
                "content": "You are a helpful assistant.",
            }
        ],
    )

    msg = cl.Message(content="")

    start_message = "Hello, I'm your 100% local ChatGPT powered by Google Deepmind's Gemma 3. How can I help you today?"

    for token in start_message:
        await msg.stream_token(token)
        await asyncio.sleep(0.005)

    await msg.send()

@cl.step(type="tool")
async def tool(input_message, image=None):

    interaction = cl.user_session.get("interaction")

    if image:
        interaction.append({"role": "user",
                            "content": input_message,
                            "images": image})
    else:
        interaction.append({"role": "user",
                            "content": input_message})

    client = ollama.AsyncClient()
    full_content = ""
    response = None
    async for chunk in await client.chat(model="gemma3:4b",
                                         messages=interaction,
                                         stream=True):
        full_content += chunk.message.content
        response = chunk

    interaction.append({"role": "assistant",
                        "content": full_content})

    response.message.content = full_content
    return response

@cl.on_message 
async def main(message: cl.Message):

    images = [file for file in message.elements if "image" in file.mime]

    if images:
        tool_res = await tool(message.content, [i.path for i in images])

    else:
        tool_res = await tool(message.content)

    msg = cl.Message(content="")

    for token in tool_res.message.content:
        await msg.stream_token(token)

    await msg.send()

What happens here:

1. User sends a message in the UI
2. Chainlit forwards the prompt to Ollama
3. Ollama runs the model
4. Response is returned to the UI

Step 3: Start the Application

Run the containers:

docker compose up -d

Step 4: Open the Chat Interface

Open your browser:

http://localhost:8000

You will now see your local AI chat interface.

Try asking :

Hi, Can you please explain virtual function in c++ ?

Popular Models You Can Run

Some popular models supported by Ollama:

ollama pull gemma:2b
ollama pull mistral:7b
ollama pull llama3
ollama pull phi3

Choose a model depending on your RAM and GPU capability.

Possible Improvements

Once your local AI is running, you can extend it further:

• Add document chat (RAG)
• Connect with LangChain
• Create AI coding assistants
• Build private enterprise chatbots

This setup is a powerful starting point for building production-ready AI applications locally.

Final Thoughts

Running LLMs locally is becoming easier thanks to tools like Ollama and Chainlit.

With just a few files and Docker Compose, you can deploy your own private AI chat interface in minutes.

No cloud. No API keys. Just your own AI running locally.

💡 If you’re interested in local AI, LLM tools, and developer productivity, this setup is a great foundation to build upon.

Thank you for being a part of the community

Before you go:

👉 Be sure to clap and follow the writer ️👏️️

👉 Follow us: **Linkedin| [Medium](https://medium.com/codetodeploy)**

👉 CodeToDeploy Tech Community is live on Discord — **Join now!**

Disclosure: This post includes affiliate and partnership links.