The Complete Guide to Running Large Language Models Locally in 2026: Hardware, Tools, and Real-World Workflows

From $351 mini PCs to 120B model: everything you need to build your own private AI stack

Not long ago, running a large language model on your own hardware was the exclusive domain of well-funded research labs and enterprise IT departments with racks of expensive server GPUs. Today, a $351 mini PC sitting on your desk can run a 35-billion-parameter model at speeds fast enough for real work. A free Kaggle GPU account is sufficient to fine-tune a capable LLM for your specific domain, after which you can deploy it privately on a Mac with no cloud subscription required. A single consumer GPU with just 12GB of VRAM can deliver near-real-time inference on a model that, by parameter count alone, sounds like it should require a data center.

This is the state of local AI in 2026: faster, cheaper, more private, and more accessible than almost anyone predicted just a few years ago.

This guide is for developers, researchers, hobbyists, and teams who want to move beyond cloud APIs and take control of their AI infrastructure. Whether you are running inference on a budget mini PC, fine-tuning a domain-specific assistant on free GPU resources, or building a fully private AI agent stack, this article will walk you through the hardware decisions, software tools, optimization techniques, and practical workflows that make it all possible. Every recommendation here is grounded in real-world demonstrations and benchmarks — not marketing specifications.

By the end, you will understand not just what to run locally, but how to run it well, and when the cloud still makes more sense.

Part One: Understanding the Fundamentals

Why Local AI? The Core Case

Before diving into hardware and tooling, it is worth being precise about why local AI matters and what problems it actually solves. The benefits fall into three clear categories.

Privacy and data sovereignty are the most compelling reasons for many teams. When you run inference locally, your prompts, your documents, and your outputs never leave your machine. For teams in medicine, law, finance, or any domain handling sensitive information, this is not a nice-to-have — it is a compliance and trust requirement. A fully local AI agent stack can operate without subscriptions, cloud services, or any external data exposure whatsoever.

Latency and reliability are the second major advantage. Local inference eliminates the round-trip to a remote API server. There is no network dependency, no rate limiting, and no service outage that can interrupt your workflow. For agentic applications that make many sequential model calls, the cumulative latency savings can be dramatic.

Cost at scale is the third factor. Cloud inference is priced per token, which is economical for occasional use but can become expensive for high-volume or always-on applications. Once you have invested in local hardware, marginal inference cost approaches zero.

The trade-off, of course, is upfront hardware investment and the engineering effort required to set up and maintain your own stack. This guide will help you minimize both.

The Fundamental Constraint: Memory, Not Compute

The single most important concept to understand before making any hardware or software decision is this: local LLM inference is almost entirely memory-bandwidth-bound, not compute-bound.

This surprises many people who assume that more CUDA cores or higher clock speeds are the primary driver of inference speed. In reality, during token generation, the bottleneck is how quickly the GPU can move model weights from memory into the compute units — not how fast those compute units can perform arithmetic once the data arrives. This means that for inference workloads, VRAM capacity and memory bandwidth matter far more than raw FLOPS.

The practical implications of this are significant:

• A GPU with more VRAM but slightly lower compute throughput will often outperform a higher-compute GPU with less VRAM on inference tasks, because it can fit more of the model on-chip and avoid slow CPU offloading.
• Quantization — reducing the numerical precision of model weights — is one of the most powerful tools available, because it directly reduces memory footprint and therefore improves how much model you can fit and how fast weights can be transferred.
• Model architecture matters as much as hardware. Mixture-of-Experts (MoE) models, which activate only a fraction of their total parameters per token, are dramatically more memory-efficient during inference than dense models of equivalent parameter count.

Keep these principles in mind as we move through hardware options and optimization techniques. They explain results that would otherwise seem impossible, like running a 35B model on 6GB of VRAM.

Quantization: The Great Equalizer

Quantization is the process of representing model weights in lower numerical precision — for example, using 4-bit integers (Q4) instead of 16-bit or 32-bit floating point. This reduces the memory footprint of a model by roughly 4x to 8x compared to full precision, at a modest cost to output quality that is often imperceptible for practical tasks.

The most common quantization formats you will encounter in the local AI ecosystem are:

GGUF (the format used by llama.cpp) supports a range of quantization levels from Q2 through Q8, with Q4 being the most popular balance of size and quality. A 70B model in Q4 quantization fits in roughly 40GB, making it feasible on a Mac Studio M4 Max with 128GB unified memory.

NF4 (4-bit NormalFloat) is used by the BitsAndBytes library for training workflows. It is specifically designed to minimize quality loss during quantization and is the format used when fine-tuning with QLoRA on limited VRAM.

Understanding quantization levels will help you make sense of model file names you encounter on Hugging Face and elsewhere. A file named Qwen3-35B-A3B-Q4_K_M.gguf, for example, tells you it is a 35B MoE model (with 3B active parameters), quantized to 4-bit with a specific quantization scheme.

Quantization Format Reference:

Format Bits/Weight Quality vs FP16 Best For Q8_0 8-bit ~99% Maximum quality, sufficient VRAM Q6_K 6-bit ~98% High quality, moderate VRAM savings Q4_K_M 4-bit mixed ~95% Best balance — recommended default Q4_K_S 4-bit small ~94% Slightly smaller than Q4_K_M Q3_K_M 3-bit mixed ~90% Very limited VRAM Q2_K 2-bit ~80% Extreme VRAM constraints only

Part Two: Choosing Your Hardware

The Hardware Decision Framework

The right hardware for local AI depends entirely on your primary use case. Inference and training have fundamentally different memory requirements, and conflating them leads to expensive mistakes.

For inference, the question is: how large a model do you need to run, and how fast does it need to be? The answers determine your VRAM requirement, which in turn determines your hardware tier.

For training and fine-tuning, the memory requirements are substantially higher. Full supervised fine-tuning of a 70B model requires approximately 300GB of memory and is effectively cloud-only for most practitioners. QLoRA fine-tuning of a 4B model, by contrast, requires about 15GB of VRAM and fits on a free Kaggle T4 GPU. Knowing where your workload falls on this spectrum is the first step.

VRAM Estimation (Q4 Quantized Models):

Model Size Approx. VRAM Required 7B 4–5 GB 13B 7–8 GB 35B ~20 GB 70B ~40 GB 120B ~65–70 GB

NVIDIA Consumer GPUs

RTX 5090 (24GB VRAM) is the current best-value NVIDIA consumer option for local inference. It can run dense 30B-class Q4 models at roughly 60–90 tokens per second. If your primary use case is inference on models up to 30B parameters and you want to stay in the NVIDIA ecosystem for CUDA compatibility, this is the card to buy.

The 24GB VRAM limit does become a genuine constraint for 70B models, which in Q4 quantization require roughly 40GB. Running a 70B model on a single RTX 5090 requires significant layer offloading to system RAM, which will substantially reduce token generation speed.

RTX 4070 (12GB VRAM) is a more affordable option that, with the right model architecture and tooling, punches well above its weight class. Using Qwen3–35B-A3B (a MoE model) with llama.cpp’s -ncmoe expert-pinning flag, a 12GB RTX 4070 can achieve 58–62 tokens per second on a 35B-class model. This is a remarkable result that would have seemed impossible on 12GB of VRAM with a dense model. The key is the MoE architecture, which activates only about 3 billion parameters per token despite having 35 billion total.

GTX 1060 (6GB VRAM) represents the absolute floor of what is practically viable for serious local inference. With careful optimization — MoE offloading, the --no-mmap and --mlock flags in llama.cpp, and aggressive memory tuning — a 35B MoE model can be run at approximately 17 tokens per second, up from a nearly unusable 3 tokens per second before optimization. This is not a recommended configuration for production use, but it demonstrates that local AI is accessible even on hardware that is several generations old.

NVIDIA Professional Cards

RTX PRO 6000 (96GB VRAM) is the professional tier option for users who need to fit very large models entirely in VRAM without quantization compromises. A single RTX PRO 6000 can comfortably fit a 70B Q4 model and support long context windows and concurrent users. The price premium over consumer cards is substantial, but for teams running inference as a service or needing maximum context length, the extra VRAM headroom is worth it.

Apple Silicon: The Unified Memory Advantage

Apple Silicon deserves special attention because its unified memory architecture fundamentally changes the hardware calculus. Unlike discrete GPU systems where VRAM and system RAM are separate pools with a slow PCIe interconnect between them, Apple Silicon uses a single shared memory pool accessible at high bandwidth by both the CPU and GPU cores. This means that a Mac with 128GB of unified memory can use all 128GB for model weights — something no discrete GPU system can match at a comparable price point.

Mac Studio M4 Max (128GB) is the practical sweet spot for 70B inference, delivering approximately 8–15 tokens per second. This is slower than a high-end NVIDIA setup, but the total cost of ownership, the silence, the power efficiency, and the seamless macOS experience make it an attractive option for many practitioners.

M4 Pro Mac mini serves as a capable entry point into the Apple Silicon ecosystem for local AI, offering strong performance for models in the 7B–30B range.

Hybrid configurations are pushing the boundaries further. A compact external AI accelerator — the “Tiiny AI box” — paired with a MacBook has been demonstrated running a 120B parameter model locally. This modular approach suggests that the ceiling for Mac-based local AI is not fixed by the Mac itself, but can be extended through external compute.

Budget and Compact Hardware

MINISFORUM UM790 Pro ($351) is perhaps the most striking data point in the current local AI landscape. This compact mini PC, built around an AMD APU, can run a 35B model locally at 20+ tokens per second with the right configuration. The key optimizations are increasing the UMA (Unified Memory Architecture) frame buffer in the BIOS to allocate more system RAM to the integrated GPU, and installing the correct AMD software stack.

This is not a configuration for power users who need maximum throughput, but for hobbyists, students, and small teams who want to experiment with large models on a tight budget, a $351 machine capable of 35B inference is a genuinely transformative option.

NVIDIA Jetson Thor vs. DGX Spark

For teams evaluating compact dedicated AI workstations, the comparison between NVIDIA’s Jetson Thor and DGX Spark is instructive. Both offer 128GB of memory, but the Jetson Thor is $1,000 cheaper than the DGX Spark. However, raw memory capacity alone does not determine practical performance — bandwidth, software ecosystem compatibility, and workload-specific optimizations all matter. Benchmarking against real workloads, rather than relying on spec sheets, is essential before committing to either platform.

Hardware Decision Summary

Use Case Recommended Hardware Expected Performance Sub-30B inference, NVIDIA RTX 5090 (24GB) 60–90 tok/s on 30B Q4 35B MoE inference, budget RTX 4070 (12GB) 58–62 tok/s 70B inference, Apple Mac Studio M4 Max (128GB) 8–15 tok/s 35B inference, extreme budget MINISFORUM UM790 Pro ($351) 20+ tok/s 70B+ inference, NVIDIA pro RTX PRO 6000 (96GB) Long context, concurrent users 120B inference, Mac hybrid Mac + Tiiny AI box Feasible QLoRA fine-tuning, 4B model Free Kaggle T4 (15GB) ~15–20 min/trial Full fine-tuning, 70B Cloud only ~300GB memory required

Part Three: The Software Ecosystem

llama.cpp: The Foundation of Local Inference

llama.cpp is the most important piece of software in the local AI ecosystem. It is a C++ inference engine for GGUF-format models that runs on virtually every platform — NVIDIA GPUs, Apple Silicon, AMD GPUs, and even CPU-only systems. Its performance, portability, and active development community have made it the backbone of local LLM inference.

Key llama.cpp flags:

Flag Purpose When to Use -ncmoe N Pin N MoE expert layers in VRAM MoE models on limited VRAM --no-mmap Disable memory-mapped file loading Low-VRAM setups with RAM spillover --mlock Lock model pages in RAM Prevent OS swapping during inference -ngl N Number of GPU layers to offload Tune GPU vs CPU layer split -c N Set context window size Control memory usage vs context length --flash-attn Enable flash attention Reduce KV cache memory for long contexts

**-ncmoe (Number of Cached MoE Experts):** This flag is the key to running MoE models efficiently on limited VRAM. By specifying how many expert layers to keep resident in VRAM — for example, -ncmoe 25 — you allow llama.cpp to pin the most frequently used experts on-chip while offloading others to system RAM. This is what makes it possible to run Qwen3-35B at 58–62 tokens per second on a 12GB GPU, reducing VRAM usage from more than 20GB to approximately 6.5–10.6GB.

**--no-mmap:** Disables memory-mapped file loading, which can improve performance and stability in low-VRAM setups where the operating system's memory management might otherwise interfere with model loading.

**--mlock:** Locks model weights in RAM, preventing the operating system from swapping them to disk. This is critical when system RAM is the primary storage for model weights that cannot fit in VRAM.

Ollama: The Developer-Friendly Layer

While llama.cpp handles the low-level inference, Ollama provides a higher-level interface that makes serving local models as easy as running a single command. Ollama wraps llama.cpp (and other backends) in a clean REST API, serves models at http://localhost:11434/v1 in an OpenAI-compatible format, and handles model downloading, caching, and lifecycle management.

For developers building applications on top of local models, Ollama’s OpenAI-compatible API means that existing code written for the OpenAI SDK can often be redirected to a local Ollama instance with minimal changes — just swap the base URL and remove the API key requirement.

Unsloth and BitsAndBytes: Making Fine-Tuning Accessible

For fine-tuning workflows, two libraries have become essential: Unsloth and BitsAndBytes.

BitsAndBytes provides the NF4 quantization that makes QLoRA fine-tuning feasible on limited VRAM. It allows a 4B model like Gemma 4 to load in 4-bit NF4 quantization and fit in roughly 8GB of memory during setup, making a free Kaggle T4 GPU sufficient for the entire fine-tuning process.

Unsloth is an optimization library that dramatically reduces memory usage and speeds up training for popular model architectures. Combined with BitsAndBytes and PEFT (Parameter-Efficient Fine-Tuning), it reduces memory consumption enough to fit training into the limited VRAM of free cloud GPU tiers.

Optuna: Automated Hyperparameter Search

Finding the right learning rate and LoRA rank for fine-tuning can significantly impact results. Optuna, a hyperparameter optimization framework, can automate this search. With just 8 short trials on a T4 GPU, Optuna can identify useful learning-rate and LoRA-rank settings in approximately 15–20 minutes — a small investment that can meaningfully improve fine-tuning outcomes.

Hermes Agent: Local AI Orchestration

For teams building AI agents rather than just running inference, Hermes Agent provides an orchestration layer that coordinates model calls, tool use, and multi-step reasoning. Paired with Gemma 4 served via Ollama, Hermes Agent is a strong foundation for fully local AI agent deployments. Self-hosted tools like Firecrawl (for web browsing) and local TTS components (for voice) can be integrated to keep every component of the agent stack on-device.

Part Four: Dense vs. MoE — A Critical Architecture Decision

The choice between dense and Mixture-of-Experts models has become one of the most consequential decisions in local AI deployment.

Dense models have all parameters active for every token. A 35B dense model requires all 35 billion parameters to be accessible in memory at inference time. In Q4 quantization, that is roughly 20GB — too large for a 12GB GPU without heavy CPU offloading.

MoE models route each token through only a subset of specialized “expert” sub-networks. Qwen3–35B-A3B, for example, has 35 billion total parameters but only approximately 3 billion active parameters per token. This means that while the full model still needs to be in memory, the compute per token is dramatically lower, and with expert-pinning tools like llama.cpp’s -ncmoe, you can keep the most-used experts in fast VRAM while the rest sit in slower system RAM.

The practical result: MoE models are dramatically more accessible on constrained hardware. They are the reason a 12GB GPU can run a “35B” model at 60 tokens per second, and why a 6GB GPU can run one at all.

Small Specialized Models vs. Large General Models

Another important consideration is whether to use a large general-purpose model or a smaller, fine-tuned specialist.

For document parsing and OCR, a 1.2B-parameter specialized model like MinerU2.5-Pro can outperform much larger general-purpose models, including commercial systems like Gemini 3 Pro and Qwen3-VL-235B, on document parsing benchmarks. It achieves a score of 95.69 on OmniDocBench v1.6 — a result that demonstrates how task-specific optimization and data engineering can outperform raw scale.

For domain-specific QA, fine-tuning a 4B model with QLoRA can improve ROUGE-L scores by 15–30% over the base model on domain-specific tasks. A fine-tuned 4B model will often outperform a general-purpose 70B model on your specific domain while being far cheaper to run locally.

The lesson: bigger is not always better for local deployment. A well-chosen, appropriately sized model — possibly fine-tuned for your domain — will often deliver better results at lower hardware cost than the largest model you can fit in memory.

Part Five: Practical Tutorials

Tutorial 1: Running a 35B MoE Model at 60 Tokens/Second on a 12GB GPU

What you need:

• A GPU with at least 12GB of VRAM (RTX 4070, RTX 3080 12GB, etc.)
• At least 32GB of system RAM
• llama.cpp installed and compiled with CUDA support
• The Qwen3–35B-A3B GGUF model file (Q4 quantization recommended)

Step 1: Install llama.cpp with CUDA support

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

Step 2: Download the model

huggingface-cli download \
  Qwen/Qwen3-35B-A3B-GGUF \
  Qwen3-35B-A3B-Q4_K_M.gguf \
  --local-dir ./models

Step 3: Run inference with expert pinning

The critical flag here is -ncmoe 25, which keeps 25 expert layers resident in VRAM. This reduces VRAM usage from more than 20GB to approximately 6.5–10.6GB, making the model fit on a 12GB card.

./build/bin/llama-cli \
  -m ./models/Qwen3-35B-A3B-Q4_K_M.gguf \
  -ngl 99 \
  -ncmoe 25 \
  --ctx-size 8192 \
  --temp 0.7 \
  -p "Explain the concept of mixture of experts in language models."

Step 4: Run as a server for API access

./build/bin/llama-server \
  -m ./models/Qwen3-35B-A3B-Q4_K_M.gguf \
  -ngl 99 \
  -ncmoe 25 \
  --ctx-size 8192 \
  --host 0.0.0.0 \
  --port 8080

You should expect 58–62 tokens per second on a 12GB RTX 4070 with this configuration. The model also supports a full 128k context window in this setup, making it viable for long-document tasks.

Troubleshooting: If you see out-of-memory errors, reduce -ncmoe from 25 to 20 or 15. Each reduction trades some performance for lower VRAM usage. If performance is lower than expected, ensure your GPU drivers are up to date and that no other GPU-intensive processes are running.

Tutorial 2: Running a 35B Model on 6GB VRAM with llama.cpp Optimization

What you need:

• A GPU with 6GB of VRAM
• At least 32GB of system RAM (the model will primarily live in system RAM)
• llama.cpp compiled with CUDA support
• A 35B MoE GGUF model (Qwen-based MoE models work best for this)

Why MoE matters here: Dense models of 35B parameters in Q4 quantization require approximately 20GB of memory — far beyond 6GB of VRAM. MoE models, however, only activate a fraction of their parameters per token. This means that with careful offloading, the GPU handles the active expert layers while system RAM holds the inactive ones. The result is that a 6GB GPU can contribute meaningfully to inference rather than being bypassed entirely.

Step 1: Configure memory flags

The --no-mmap flag prevents the OS from using memory-mapped file access, which can cause unpredictable performance when the model is larger than available RAM. The --mlock flag locks the model weights in RAM, preventing them from being swapped to disk mid-inference:

./build/bin/llama-cli \
  -m ./models/Qwen-MoE-35B-Q4_K_M.gguf \
  -ngl 10 \
  --no-mmap \
  --mlock \
  --ctx-size 2048 \
  -p "Your prompt here"

Note that -ngl 10 offloads only 10 layers to the GPU, leaving the rest on CPU/RAM. Experiment with this value — higher values improve speed until you hit the VRAM ceiling, at which point performance degrades sharply.

Step 2: Tune for your system

The optimal -ngl value depends on your specific model and GPU. Start at 10 and increase by 5 until you see out-of-memory errors, then back off by 5. On a 6GB GPU, you might find the sweet spot around 10–15 layers.

Expected performance: With the right configuration, token generation speed can improve from approximately 3 tok/s (naive setup) to approximately 17 tok/s (optimized) — a nearly 6x improvement from tuning alone, with no hardware change.

Tutorial 3: Fine-Tuning Gemma 4 on a Free Kaggle GPU and Deploying Locally on a Mac

This tutorial covers the complete workflow from fine-tuning a domain-specific model on free cloud resources to deploying it privately on Apple Silicon.

What you need:

• A free Kaggle account (provides T4 GPU access)
• An Apple Silicon Mac (M1 or later, with at least 16GB unified memory)
• Ollama installed on your Mac
• A domain-specific dataset in question-answer format

Why this workflow matters: Fine-tuning can improve ROUGE-L scores by 15–30% on domain-specific QA tasks compared to the base model. For teams that need a model specialized in their domain — medical terminology, legal language, proprietary product knowledge — this improvement can be the difference between a useful tool and an unreliable one. And by exporting to GGUF and running locally, you get that specialized capability with complete data privacy.

Step 1: Set up the Kaggle notebook

Create a new Kaggle notebook and enable GPU acceleration (T4). Install the required libraries:

!pip install -q unsloth bitsandbytes peft transformers datasets optuna

Step 2: Load Gemma 4 in 4-bit NF4 quantization

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="google/gemma-4-4b-it",
    max_seq_length=2048,
    dtype=None,          # Auto-detect
    load_in_4bit=True,   # NF4 quantization via BitsAndBytes
)

Step 3: Apply LoRA adapters

Rather than fine-tuning all model weights (which would require far more memory), QLoRA trains small adapter layers that are added to the frozen base model:

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=42,
)

Step 4: Use Optuna for hyperparameter search

Rather than guessing the optimal learning rate and LoRA rank, use Optuna to run 8 short trials and find good settings automatically. This takes approximately 15–20 minutes on a T4:

import optuna

from transformers import TrainingArguments
from trl import SFTTrainer

def objective(trial):
    lr = trial.suggest_float("lr", 1e-5, 3e-4, log=True)
    r = trial.suggest_categorical("r", [8, 16, 32])

    model = FastLanguageModel.get_peft_model(
        base_model, r=r,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
        lora_alpha=r, lora_dropout=0, bias="none",
        use_gradient_checkpointing="unsloth",
    )

    training_args = TrainingArguments(
        output_dir="./trial_output",
        num_train_epochs=1,
        per_device_train_batch_size=2,
        learning_rate=lr,
        fp16=True,
        logging_steps=10,
        max_steps=50,
    )

    trainer = SFTTrainer(
        model=model,
        tokenizer=tokenizer,
        train_dataset=train_dataset,
        args=training_args,
    )

    trainer.train()
    metrics = trainer.evaluate()
    return metrics["eval_loss"]

study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=8)
print("Best params:", study.best_params)

Step 5: Full fine-tuning with best parameters

best_lr = study.best_params["lr"]
best_r = study.best_params["r"]

training_args = TrainingArguments(
    output_dir="./fine_tuned_gemma4",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=best_lr,
    fp16=True,
    logging_steps=25,
    save_strategy="epoch",
    warmup_ratio=0.1,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    args=training_args,
    dataset_text_field="text",
    max_seq_length=2048,
)

trainer.train()

Step 6: Export to GGUF for local deployment

# Save merged model
model.save_pretrained_merged(
    "merged_gemma4",
    tokenizer,
    save_method="merged_16bit",
)

# Convert to GGUF (run in the llama.cpp directory)
# !python convert_hf_to_gguf.py merged_gemma4 --outtype q4_k_m --outfile gemma4_finetuned.gguf

Step 7: Deploy on your Mac with Ollama

Download the GGUF file from Kaggle and place it in a local directory. Create an Ollama Modelfile:

FROM ./gemma4_finetuned.gguf
PARAMETER temperature 0.7
PARAMETER num_ctx 4096
SYSTEM "You are a specialized assistant with expertise in [your domain]. Answer questions accurately and concisely."

Create and run the model:

ollama create my-domain-assistant -f Modelfile
ollama run my-domain-assistant

Expected performance on Apple Silicon: A quantized GGUF export runs on an M4 Pro Mac at approximately 35–50 tokens per second with under 1 second to first token. This is fast enough for interactive use and represents a completely private, domain-specialized AI assistant with no ongoing cloud costs.

Tutorial 4: Building a Fully Local Private AI Agent

What you need:

• A Mac or Linux machine with sufficient RAM for your chosen model
• Ollama installed
• Node.js (for Hermes Agent)
• Docker (for self-hosted tools like Firecrawl)

Step 1: Install and configure Ollama

# macOS
brew install ollama

# Start the Ollama service
ollama serve

# Pull Gemma 4
ollama pull gemma4

Verify the model is serving correctly:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4",
    "messages": [{"role": "user", "content": "Hello, are you running locally?"}]
  }'

Ollama serves at http://localhost:11434/v1 in an OpenAI-compatible format, which means any application built for the OpenAI API can be redirected to your local model by changing the base URL.

Step 2: Set up Hermes Agent

git clone https://github.com/restarone/hermes-agent
cd hermes-agent
npm install

Configure Hermes to use your local Ollama endpoint:

{
  "llm": {
    "provider": "openai-compatible",
    "baseUrl": "http://localhost:11434/v1",
    "model": "gemma4",
    "apiKey": "not-required"
  },
  "tools": {
    "browser": {
      "provider": "firecrawl",
      "baseUrl": "http://localhost:3002"
    }
  }
}

Step 3: Self-host Firecrawl for private web browsing

git clone https://github.com/mendableai/firecrawl
cd firecrawl
docker-compose up -d

Step 4: Add local TTS for voice interaction

pip install piper-tts

Configure Hermes to use Piper for voice output in the agent configuration.

Step 5: Run your first agent task

npm start -- --task "Research the latest developments in local AI hardware and summarize the key findings"

The agent will use Gemma 4 via Ollama for reasoning, Firecrawl for web browsing, and Piper for voice output — all running locally, with no data leaving your machine.

Tutorial 5: Optimizing a Budget Mini PC for 35B Inference

What you need:

• MINISFORUM UM790 Pro or similar AMD APU mini PC
• At least 32GB of DDR5 RAM (RAM is shared with the GPU)
• llama.cpp compiled for AMD (ROCm or Vulkan backend)

Step 1: Increase the UMA Frame Buffer in BIOS

This is the single most important optimization for AMD APU systems. The UMA frame buffer determines how much system RAM is reserved for the integrated GPU. By default, this is often set conservatively. Increasing it gives the GPU more memory to work with:

1. Restart and enter BIOS (usually by pressing Delete or F2 during boot)
2. Navigate to Advanced → AMD CBS → NBIO → GFX Configuration
3. Set UMA Frame Buffer Size to 8GB or 16GB (depending on your total RAM)
4. Save and restart

Step 2: Install AMD ROCm or use Vulkan backend

# Compile llama.cpp with Vulkan support (works on Windows and Linux)
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j$(nproc)

Step 3: Run the 35B model

./build/bin/llama-cli \
  -m ./models/Qwen3-35B-A3B-Q4_K_M.gguf \
  -ngl 99 \
  -ncmoe 20 \
  --ctx-size 4096 \
  --temp 0.7 \
  -p "Your prompt here"

With the UMA frame buffer increased and the correct drivers installed, you should see 20+ tokens per second on a 35B model — a remarkable result for a $351 machine.

Tutorial 6: Using Ollama’s OpenAI-Compatible API in Your Applications

One of the most practical aspects of Ollama is that it exposes an OpenAI-compatible API, meaning you can swap out cloud API calls for local inference with minimal code changes.

Python example using the OpenAI SDK:

from openai import OpenAI

# Point to local Ollama instead of OpenAI
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Required by the SDK but not used by Ollama
)

response = client.chat.completions.create(
    model="qwen2.5:32b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantization in LLMs in plain English."}
    ]
)

print(response.choices[0].message.content)

Building a local RAG pipeline:

from openai import OpenAI
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import OllamaEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

# Load and index local documents
loader = DirectoryLoader("./docs", glob="**/*.txt")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

# Create local vector store using Ollama embeddings
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(chunks, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

def ask(question: str) -> str:
    context_docs = retriever.invoke(question)
    context = "\n\n".join(d.page_content for d in context_docs)

    response = client.chat.completions.create(
        model="qwen2.5:32b",
        messages=[
            {
                "role": "system",
                "content": "Answer questions using only the provided context."
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {question}"
            }
        ]
    )

    return response.choices[0].message.content

print(ask("What are the key findings in the Q3 report?"))

This entire pipeline — document loading, embedding, retrieval, and generation — runs locally with no external API calls.

Part Six: When to Use the Cloud

Local AI is not always the right answer. Understanding when the cloud is the better choice will save you money and frustration.

Training at Scale

The memory requirements for training are substantially higher than for inference. Full supervised fine-tuning of a 70B model requires approximately 300GB of memory — a configuration that is effectively cloud-only for most practitioners. Even QLoRA fine-tuning of models larger than about 13B begins to strain consumer hardware. For serious training workloads, cloud B200 instances are often cheaper and faster than buying expensive local pro hardware.

Always-On Production Workloads

If you need to serve inference to many concurrent users around the clock, the economics of local hardware versus cloud can shift. A cloud B200 instance provides massive throughput and requires no upfront capital expenditure. For large-scale or always-on production work, cloud is often the more practical and economical option.

Advanced Training Techniques

Reinforcement learning from human feedback (RLHF) and other advanced training techniques for 70B-class models typically require cloud GPUs or professional cards. The memory and compute requirements exceed what consumer hardware can provide.

The Hybrid Approach

For many teams, the optimal strategy is hybrid: use local hardware for inference, experimentation, and privacy-sensitive workloads, and use cloud GPUs for training runs and large-scale fine-tuning. The workflow demonstrated in Tutorial 3 — fine-tuning on a free Kaggle T4 and deploying locally on a Mac — is a practical example of this hybrid approach.

Part Seven: Emerging Trends and What to Watch

Modular Compute

The demonstration of a 120B model running on a MacBook paired with a compact external accelerator points toward a future where local AI capability is not fixed by the machine you buy, but can be extended modularly. This is analogous to how external GPUs (eGPUs) extended laptop graphics capability — but purpose-built for AI inference workloads.

Open-Source Catching Up to Commercial Systems

The MinerU2.5-Pro result — a 1.2B open-source model outperforming commercial systems on document parsing — is part of a broader pattern. Task-specific open-source models, trained with careful data engineering rather than simply scaled to more parameters, are increasingly competitive with or superior to much larger commercial offerings on specific tasks. This trend has significant implications for local AI: as open-source models improve, the case for cloud-hosted proprietary models weakens.

The CUDA Ecosystem vs. Alternatives

CUDA remains the most mature ecosystem for serious training and advanced techniques, and NVIDIA’s dominance in the professional AI hardware market is not threatened in the near term. However, Apple Silicon’s unified memory architecture and AMD’s improving ROCm stack are making non-NVIDIA options increasingly viable for inference workloads. The Vulkan backend in llama.cpp provides a cross-platform path that works on AMD, Intel, and even some mobile GPUs.

Conclusion: Building Your Local AI Stack

The local AI landscape in 2026 offers a genuine spectrum of options, from a $351 mini PC to a Mac Studio with 128GB of unified memory to hybrid configurations that push the boundaries of what personal hardware can do. The right choice depends on your specific workload, budget, and priorities.

Here is a practical framework for making your decision:

Start with your model size requirement. What is the smallest model that will deliver acceptable quality for your use case? Fine-tuning a smaller model for your domain may outperform a larger general-purpose model. Specialized small models can outperform much larger commercial systems on specific tasks.

Match hardware to memory requirements. Once you know your model size, calculate the memory requirement (roughly: parameters × 0.5 bytes for Q4 quantization). Choose hardware that can fit this in VRAM or unified memory, with headroom for context.

Choose MoE models when hardware is constrained. If your VRAM is limited, MoE models give you access to much larger parameter counts at a fraction of the memory cost. A 12GB GPU running a 35B MoE model at 60 tokens per second is a better choice than the same GPU struggling with a 13B dense model.

Use the right tools. llama.cpp with appropriate flags, Ollama for serving, Unsloth and BitsAndBytes for fine-tuning — these tools are what make local AI practical. The hardware matters, but the software stack matters just as much.

Keep training in the cloud. Unless you have professional-grade hardware, use free or low-cost cloud GPU resources for fine-tuning and run the resulting model locally. The hybrid workflow is the sweet spot for most practitioners.

Local AI is no longer a compromise. With the right hardware, the right models, and the right tools, you can run capable, fast, private AI entirely on your own hardware — and the gap between local and cloud is closing faster than anyone expected.