Build Your Own Private AI Assistant: Local LLMs + Free Cloud Models + Remote Access via Telegram

A weekend project that gives you a self-hosted ChatGPT-style assistant — private by default, with a fallback to free cloud models (not so private), accessible from anywhere, and controllable from your phone.

Disclaimer: This article is provided for educational and informational purposes only and does not constitute professional security, legal, financial, IT, or engineering advice. All software, code samples, commands, configurations, and examples are provided “as is” without warranties of any kind, express or implied. The author assumes no responsibility or liability for any data loss, security incidents, service interruptions, account compromises, financial losses, hardware damage, or other consequences arising from the use of the information presented. Readers are solely responsible for testing, securing, and validating any configuration before deploying it in their own environments.

Why Build This

Cloud AI assistants are convenient, but every message you send goes through someone else’s servers. For journaling, personal notes, financial planning, or anything sensitive, that’s a real tradeoff.

At the same time, fully local AI has its own limits — local models don’t know about anything that happened after their training cutoff, and consumer GPUs can’t run the largest, smartest models.

This guide walks through a hybrid setup that gets you the best of both:

• A local LLM running on your own hardware (private, always available, no internet required)
• Free-tier cloud models as a fallback for current-events questions or when you need more horsepower
• A unified chat interface where you pick which “brain” to talk to from a simple dropdown
• Remote access from your phone over an encrypted private network — no port forwarding, no exposing anything to the public internet
• Remote on/off control via a Telegram bot, so you can toggle remote access from anywhere

Everything here runs in Docker containers, so the core setup is nearly identical across Windows, Linux, and macOS — the differences are mostly in a handful of OS-specific commands, which are called out explicitly below.

Architecture Overview

The setup has five pieces, each doing one job:

• Ollama runs open-weight language models locally using your GPU (or CPU, if no GPU is available)
• LiteLLM is a lightweight proxy that exposes a single OpenAI-compatible API endpoint, routing requests to either your local models or a cloud provider’s free tier
• Open WebUI is the chat front-end — looks and feels like a typical AI chat app, with a model-picker dropdown
• Tailscale creates a private mesh network between your devices, so your phone can reach your home server securely without opening any ports to the internet
• A small Telegram bot lets you remotely enable/disable that private network connection from your phone, useful if you want an extra layer of control over when your home server is reachable

The flow, in order:

1. Your phone or browser opens Open WebUI
2. Open WebUI sends your message to LiteLLM
3. LiteLLM routes it to either your local Ollama model, or to a free model on OpenRouter — based on what you picked, with automatic fallback if one is unavailable

Prerequisites

• A computer that stays on most of the time (a desktop, home server, or always-on laptop) — this will run the “brains”
• Docker and Docker Compose installed
• A GPU is optional but significantly speeds up local inference. 6–8GB of VRAM is enough for small-to-medium models (3B–8B parameters)
• A free OpenRouter account for cloud fallback models
• A smartphone for remote access

Step 1: Set Up the Core Stack with Docker Compose

Create a project folder and a docker-compose.yml file:

Windows (PowerShell):

New-Item -ItemType Directory `
  -Path .\ai-stack -Force
cd .\ai-stack

Linux / macOS:

mkdir -p ~/ai-stack
cd ~/ai-stack

Create docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    tty: true
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_FLASH_ATTENTION=1
      - OLLAMA_KV_CACHE_TYPE=q8_0
      - OLLAMA_NUM_PARALLEL=2
      - OLLAMA_MAX_LOADED_MODELS=1
      - OLLAMA_KEEP_ALIVE=30m
    healthcheck:
      test: [
        "CMD-SHELL",
        "curl -f http://localhost:11434 || exit 1"
      ]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    # GPU acceleration (NVIDIA only).
    # Remove this block for CPU-only
    # or non-NVIDIA setups.
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: unless-stopped
    ports:
      - "4000:4000"
    volumes:
      - ./litellm-config.yaml:/app/config.yaml
    command: [
      "--config", "/app/config.yaml",
      "--port", "4000"
    ]
    environment:
      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
    depends_on:
      ollama:
        condition: service_healthy

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    volumes:
      - open-webui:/app/backend/data
    ports:
      - "3000:8080"
    environment:
      - OPENAI_API_BASE_URL=http://litellm:4000
      - OPENAI_API_KEY=${LITELLM_MASTER_KEY}
      - WEBUI_AUTH=true
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
      - CORS_ALLOW_ORIGIN=*
    depends_on:
      - litellm

volumes:
  ollama:
  open-webui:

Note on GPU support: On Linux, NVIDIA GPU passthrough to Docker requires the NVIDIA Container Toolkit installed separately. On Windows, Docker Desktop with WSL2 handles this automatically once NVIDIA drivers support WSL2 GPU passthrough. On macOS (Apple Silicon), Docker cannot access the GPU directly — for best performance, consider running Ollama natively on macOS instead of in a container, and pointing the other services at [http://host.docker.internal:11434.](http://host.docker.internal:11434.)

Step 2: Secure Your Secrets

Never hardcode API keys or secret strings directly into your compose file. Use a .env file instead.

Generate random secrets:

Windows (PowerShell):

$webuiKey = -join (
  (1..32) | ForEach-Object {
    '{0:x2}' -f (Get-Random -Maximum 256)
  }
)

$liteLLMKey = "sk-" + (-join (
  (1..24) | ForEach-Object {
    '{0:x2}' -f (Get-Random -Maximum 256)
  }
))

Linux / macOS:

WEBUI_KEY=$(openssl rand -hex 32)
LITELLM_KEY="sk-$(openssl rand -hex 24)"

Create .env:

OPENROUTER_API_KEY=your-openrouter-key-here
LITELLM_MASTER_KEY=sk-your-generated-key-here
WEBUI_SECRET_KEY=your-generated-key-here

Add .env to .gitignore so it never ends up in version control:

echo ".env" >> .gitignore

Step 3: Pull a Local Model

Pick a model that fits your hardware. For a GPU with 6–8GB VRAM, an 8-billion-parameter model is a good starting point.

docker exec -it ollama \
  ollama pull llama3.1:8b

Other good options depending on your needs and hardware: smaller 3B models for faster responses on modest hardware, or larger models if you have more VRAM to spare.

Run this anytime to see what you’ve downloaded:

docker exec -it ollama ollama list

Step 4: Configure LiteLLM Routing

Create litellm-config.yaml. This defines which models appear in your chat dropdown, and sets up automatic fallback if a model is unavailable or rate-limited:

model_list:
  - model_name: local-chat
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://ollama:11434

  - model_name: cloud-chat-small
    litellm_params:
      model: >-
        openrouter/meta-llama/
        llama-3.2-3b-instruct:free
      api_key: "os.environ/OPENROUTER_API_KEY"

  - model_name: cloud-chat-large
    litellm_params:
      model: >-
        openrouter/openai/
        gpt-oss-120b:free
      api_key: "os.environ/OPENROUTER_API_KEY"

router_settings:
  fallbacks:
    - cloud-chat-large:
        - cloud-chat-small
        - local-chat
    - cloud-chat-small:
        - local-chat

general_settings:
  master_key: "os.environ/LITELLM_MASTER_KEY"

Tip: Free-tier model IDs on OpenRouter change over time as providers add and retire models. Before relying on a specific model ID, check the current list at openrouter.ai/api/v1/models and filter for :free suffixes.

Note: the model: lines above use YAML's >- folded scalar to split a long model ID across lines for readability here — when you create your own file, it's simplest to just write the full ID on one line (e.g. model: openrouter/meta-llama/llama-3.2-3b-instruct:free).

Step 5: Launch Everything

docker compose up -d

Check that everything started cleanly:

docker compose logs litellm

Open http://localhost:3000 in your browser. The first account you create becomes the admin account.

You should now see a model dropdown with your local model and the two cloud fallback options. Pick local for anything private, and a cloud option when you need current information or more reasoning power.

Step 6: Lock Down Access and Add Users

Open WebUI includes built-in user management:

• Admin Panel → Settings → General: set “Default User Role” to pending so new signups require approval, or disable signups entirely once your users are set up
• Admin Panel → Users → Groups: create groups and restrict access to specific models per group — useful if you want to reserve a larger model for yourself while giving other household members access only to smaller, faster ones

Step 7: Secure Remote Access with a Private Mesh Network

Rather than exposing port 3000 to the public internet (a significant security risk), use a mesh VPN tool like Tailscale (or alternatives like ZeroTier or Netbird) to create a private network between your devices.

Install on your server:

Windows:

winget install tailscale.tailscale

Linux:

curl -fsSL https://tailscale.com/install.sh | sh

macOS:

brew install tailscale

(or download from the Mac App Store)

Install on your phone: search “Tailscale” in your app store, sign in with the same account.

Enable HTTPS for your service:

tailscale serve --bg http://localhost:3000

Check your private HTTPS address:

tailscale serve status

You’ll get a stable address like https://your-device-name.your-tailnet.ts.net — this works from anywhere, doesn't change when your home IP changes, and is encrypted end-to-end.

Add it to your phone’s home screen:

Open the HTTPS address in Chrome (Android) or Safari (iOS), then use “Add to Home Screen” / “Install App” — because it’s served over HTTPS, your phone will offer a proper app-like install with no address bar, behaving like a native app.

A note on VPN conflicts: Most phones only allow one active VPN connection at a time. If you also use a commercial VPN app for general privacy, you’ll need to disconnect it before connecting to your mesh network, and vice versa.

Step 8: Optional — Remote Control via Telegram

If you want to toggle your mesh network connection remotely (for example, turning it off when you don’t want your home server reachable, and back on when you do), a small Telegram bot can do this with three commands: enable, disable, and status.

Create a bot:

1. Message @BotFather on Telegram, send /newbot, follow the prompts, and save the token it gives you
2. Message @userinfobot to get your numeric Telegram user ID (used to restrict who can control the bot)

A minimal bot script (bot.py):

import subprocess
import logging
from logging.handlers import (
    RotatingFileHandler
)
from telegram import Update
from telegram.ext import (
    Application,
    CommandHandler,
    ContextTypes,
)

BOT_TOKEN = "your-bot-token"
ALLOWED_USER_ID = 123456789  # your ID

handler = RotatingFileHandler(
    "bot.log",
    maxBytes=100 * 1024 * 1024,
    backupCount=2,
)
logging.basicConfig(
    level=logging.INFO,
    handlers=[handler, logging.StreamHandler()],
    format="%(asctime)s [%(levelname)s] %(message)s",
)
logger = logging.getLogger(__name__)

async def check_auth(update: Update) -> bool:
    user_id = update.effective_user.id
    if user_id != ALLOWED_USER_ID:
        logger.warning(
            f"Unauthorized attempt by {user_id}"
        )
        await update.message.reply_text(
            "Unauthorized."
        )
        return False
    return True

async def enable(update, context):
    if not await check_auth(update):
        return
    subprocess.run(
        ["tailscale", "up"], check=False
    )
    await update.message.reply_text(
        "Network enabled."
    )

async def disable(update, context):
    if not await check_auth(update):
        return
    subprocess.run(
        ["tailscale", "down"], check=False
    )
    await update.message.reply_text(
        "Network disabled."
    )

async def status(update, context):
    if not await check_auth(update):
        return
    result = subprocess.run(
        ["tailscale", "status"],
        capture_output=True,
        text=True,
    )
    await update.message.reply_text(
        result.stdout[:1000] or "No output"
    )

app = Application.builder() \
    .token(BOT_TOKEN) \
    .build()

app.add_handler(
    CommandHandler(["enable", "e"], enable)
)
app.add_handler(
    CommandHandler(["disable", "d"], disable)
)
app.add_handler(
    CommandHandler(["status", "s"], status)
)

app.run_polling()

Install the dependency:

pip install python-telegram-bot \
  --break-system-packages

Run it persistently:

• Linux/macOS: use a systemd service or launchd agent to keep it running and restart on crash
• Windows: use Task Scheduler with “Run whether user is logged on or not” and a restart-on-failure policy; use pythonw.exe instead of python.exe to avoid a visible console window

Once running, send /e, /d, or /s to your bot from anywhere to enable, disable, or check your mesh network connection.

What You End Up With

• A private chat interface accessible from your phone, anywhere, over an encrypted connection
• Local models for anything sensitive — nothing leaves your network
• Free cloud models as a fallback for current-events questions or heavier reasoning
• Multi-user support with per-group model access control
• Remote on/off control over connectivity via Telegram

Total cost: $0 beyond electricity, assuming you’re using free-tier cloud models and free mesh networking. Total time: a focused afternoon, more if you’re new to Docker.

A Note on Privacy

Local models keep everything on your hardware — full privacy. Free-tier cloud models route your prompts through third-party infrastructure, and many free tiers permit the underlying provider to use your data for training. Treat anything sent to a free cloud model as non-private: avoid personal identifying information, financial details, or anything sensitive. Use local models for that, and reserve cloud fallback for general or already-public information.

Have you built something similar? I’d love to hear what models and hardware combinations have worked well for you — drop a comment below.

Disclaimer: This article is provided for educational and informational purposes only and does not constitute professional security, legal, financial, IT, or engineering advice. All software, code samples, commands, configurations, and examples are provided “as is” without warranties of any kind, express or implied. The author assumes no responsibility or liability for any data loss, security incidents, service interruptions, account compromises, financial losses, hardware damage, or other consequences arising from the use of the information presented. Readers are solely responsible for testing, securing, and validating any configuration before deploying it in their own environments.