Coding with AI Agents — Offline, Free, and Without Excuses: Part 3 — “The Upgrade”

If you missed Part 1 and Part 2, they cover the reasoning behind a free offline AI coding setup and the initial stack using LM Studio and Continue VSCode plugin. This part is about what changed — the models, the runtime, and the tools.

When I wrote Part 2 two months ago, every decision was a negotiation with available memory — quantization level, context window size, model size versus speed. I was making it work, but always within limits.

Since then I moved from an M1 Pro to an M5 Max. But honestly, that is not the main story. In two months the models have gotten smaller and faster, and the tooling has matured noticeably. What required serious hardware compromises is already a different conversation.

Before I go further — a necessary disclaimer. Local models are not a replacement for Claude, GPT-4o, or Gemini. If you are expecting the same quality, you will be disappointed. The gap is real, it is closing, but it has not closed. What local gives you is a different set of tradeoffs: privacy, zero cost, no rate limits, and the ability to run 24/7 without a bill. For the right tasks, that is a compelling trade. For the wrong ones, it is a frustrating compromise.

This article is about where that line is today for me. What to run, why, and how to set it up.

Photo by Daniele Levis Pelusi on Unsplash

The Why Has Changed

In Part 1, I wrote about two reasons for going local: working offline on planes, and making the tools accessible to people the subscription pricing was quietly excluding.

Both still hold. But there is a third reason now.

The direction this ecosystem is heading is not a smarter autocomplete. It is automation. Agents that run continuously, execute tasks, iterate on results, and feed their output into other agents. A coding pipeline that works while you sleep. A research loop that runs for six hours while you are in meetings. A virtual team that does not clock out.

Run that kind of workload against a cloud API and the token costs scale brutally. A 24/7 agentic pipeline hitting GPT-4o or Claude Sonnet at volume is not a $20/month problem — it is a pricing conversation you do not want to have with yourself at 3am when you check the usage dashboard.

Run it locally and the constraint disappears. The only bill is electricity.

Why the Hardware Matters More Than You Think

Local AI is, at its core, a memory problem.

Cloud-based models like GPT-4o or Claude run on clusters with hundreds of gigabytes of specialized GPU memory. When you run a model locally, everything — the model weights, the Key-Value cache, the active context — has to fit in RAM. On most machines, this forces brutal compromises: smaller models, lower quantization, shorter context windows.

Apple Silicon changes the equation because it uses unified memory — the CPU, GPU, and Neural Engine share the same memory pool, with no copying data between components that bottlenecks conventional hardware. Apple explicitly calls out large language models as a primary use case for the M5 family — and the numbers back that up in practice.

On my M1 Pro, the realistic ceiling was Qwen2.5-Coder 32B, and it ran slowly enough to notice.

The Ollama MLX Story

When I wrote Part 2, one of Ollama’s real weaknesses on Mac was its llama.cpp Metal backend. LM Studio had been using MLX since October 2024 — Apple's native machine learning framework — which gave it a real speed advantage on Apple Silicon for over a year.

On March 30, 2026, Ollama closed that gap. Version 0.19 swapped in MLX for Apple Silicon, bringing a significant improvement to both generation speed and prompt processing. The performance argument for LM Studio on Mac no longer holds.

Choosing the Models

This is the part that took the most time, and where I want to be specific rather than vague.

The question is not “which model is best.” It is “which model is best for which task.” Those are different questions, and conflating them is how people end up with one enormous model doing everything badly.

I landed on two models for the developer workflow:

Devstral Small 2 (24B) — for Agentic Coding

Devstral is built by Mistral AI specifically for software engineering agents. Not for chat. Not for general assistance. For the specific task of exploring codebases, editing multiple files, and powering software engineering agents.

At 24B parameters with a 256K context window, it fits comfortably in memory alongside other running processes. That context window is double what most competing models at this size offer — meaning it can hold an entire project structure in memory without chunking.

On SWE-bench Verified — a benchmark that gives a model real GitHub issues from real codebases and asks it to fix them — Devstral Small 2 scores 68%, placing it firmly among models up to five times its size. I am running it as the active coding agent — the model that edits files, runs in feedback loops, and handles multi-step implementation tasks.

The Apache 2.0 license means no commercial restrictions. You can use this in production workflows, in client projects, in anything.

Qwen3.6–27B (27B) — for Thinking and Review

Released by Alibaba on April 22, 2026, Qwen3.6–27B is a dense model — meaning all 27 billion parameters activate on every inference pass. This sounds less efficient than Mixture-of-Experts architectures, but it has real advantages: simpler deployment, more consistent output, and no routing complexity.

It scores 77.2% on SWE-bench Verified, edging out Devstral on raw coding quality. But it has something more important: a thinking mode. When you enable it, the model reasons through the problem before answering — slower, but noticeably better on architectural decisions, system design questions, and anything where the answer requires weighing tradeoffs rather than pattern-matching to known code.

I use this model when I want to think out loud about a large codebase, not when I want something to go and make changes. The 262K context window means it can hold a significant slice of a real project in a single prompt.

The two models together use roughly 40GB of RAM*.

The Installation

No GUI required. Three commands.

brew install ollama
brew services start ollama

Then pull the models:

ollama pull devstral:24b
ollama pull qwen3.6:27b

Both are in the Ollama registry. Both download cleanly. devstral:24b is about 15GB, qwen3.6:27b is about 17GB. Start them in the background and come back when they finish.

Verify they are loaded:

ollama list

You should see both. Then verify the API is responding:

curl http://localhost:11434/v1/models

If you get a JSON response listing your models, you are ready.

Connecting to VS Code — Why I Switched from Continue to Cline

In Part 2 I used Continue.

Continue is solid for autocomplete and basic chat. But when I started using it for real agentic tasks — multi-file edits, feedback loops, iterating on implementation — the experience felt flat. Responses appeared in a panel. I manually applied changes. The loop was broken.

I switched to Cline and the difference was immediate.

Cline acts. When you give it a task, it reads the relevant files, proposes changes with a full diff, waits for your approval, applies the edit, runs the result, reads the output, and continues — all in a visible, auditable loop. You see exactly what it is doing at every step. You approve or reject each change. The feedback is continuous rather than fragmented.

This is the difference between a tool that assists and a tool that works. Continue helps you write code. Cline writes code while you watch and steer.

Install it from the VS Code marketplace — search for Cline.

Configuration

Cline has a built-in Ollama provider. Open Cline settings (gear icon in the panel) and set:

After registering you will be able to access:

Now lets point It to locall Ollama

Model → devstral:24b Model Context Window → 262144 (Devstral's full 256K) Request Timeout → 120000 (local models need more time than cloud APIs)

The Plan / Act Split

This is where it gets interesting. Cline supports separate models for Plan mode and Act mode. Enable “Use different models for Plan and Act modes” in settings, then configure:

1. Plan mode → qwen3.6:27b— Qwen thinks through the problem, proposes an approach, reasons about tradeoffs
2. Act mode → devstral:24b— Devstral executes the plan, edits files, runs commands, iterates

In practice: switch to Plan, describe what you want to build or change, let Qwen reason through the architecture. Switch to Act, Devstral implements it. Two models, two roles, one coherent workflow — entirely local, entirely free.

The Honest Performance Picture

Generation speed on the M5 Max delivers roughly 35 tok/s on a 70B model at Q4 — fast enough that you are reading as it writes, not waiting for it to finish.

The more interesting performance story is context. Devstral’s 256K window means you can feed it an entire monorepo’s project graph*, the relevant library code, and a specific issue description, and it holds all of it. On my previous machine that kind of prompt would overflow the window. Now it fits.

What has not changed: the models are not as capable as the frontier cloud options. For complex reasoning and tasks requiring deep domain knowledge, cloud models still produce notably better results. If you are doing something genuinely novel and architecturally complex, Claude Sonnet or GPT-4o will still outperform what you can run locally.

What has changed: the gap between open-source and proprietary models has effectively closed for software engineering tasks. For most daily coding tasks — code completion, explanation, refactoring, and test generation — local models now handle them at a quality level that would have required cloud-only territory eighteen months ago. They are fast, free, private, and increasingly capable.

I do not Know Exactly Where this Goes

The direction is clear — multiple models, specialized roles, an orchestration layer that routes tasks to the right model and passes results between them. A local virtual team that can plan, implement, test, and iterate without cloud dependency. That is the vision, and the infrastructure described in this article is the foundation it would run on.

Whether I get there, and how long it takes, I cannot promise. This space moves fast, the tooling is still maturing, and building something genuinely useful out of these pieces is harder than assembling them. I am figuring it out as I go, the same as everyone else.

If it comes together, that is Part 4. If it does not, I will write honestly about why.

For now: get Ollama running, pull the two models, install Cline, and spend an hour with your own codebase. The experience of having a capable coding agent running entirely on your own hardware, offline, at no cost, is different enough from the theoretical description that it is worth just doing.

If your setup looks different — different models, different runtime, different editor — I want to hear about it in the comments.