One Update Made My Local AI 90% Faster. The 90% Disappears the Moment You Stop Coding.

I updated Ollama on a Tuesday night, restarted my coding agent, and watched it tear through a refactor at roughly twice the speed it managed the week before. My first reaction was the uncritical kind: finally, a free win, no new hardware, no new model. My second reaction, about ten minutes later, was to open an ordinary chat window, ask the same model to help me rewrite a paragraph of this article, and discover that the speedup had quietly evaporated.

That gap between the two experiences is the whole story, and it is the part the announcements skip over. Ollama 0.31 makes Gemma 4 generate tokens nearly ninety percent faster on Apple Silicon. That figure is real, it is measured, and it happens to have been recorded on the exact machine I use, a MacBook Pro with an M5 Max. It is also conditional in a way that matters enormously depending on what you actually do with a local model. I want to walk through why the number is genuine, why it is not universal, and how to tell in advance whether it will show up for your work or sail right past it.

One command, and Gemma 4 nearly doubled

The mechanics of getting the speedup are almost insultingly simple, which is part of why it spread so fast. You update to Ollama 0.31 or later, you re-pull the model so you get the version that ships with the new machinery, and you run it. There is no flag to set, no configuration file to edit, no environment variable to remember. The relevant command is a single line, ollama pull gemma4:12b-mlx, and from that point the acceleration is on by default.

On the coding-agent benchmark Ollama published, Gemma 4 12B went from about 50 tokens per second to about 95 on an M5 Max. I saw something in that neighborhood the first night, running a real agent against a real codebase rather than a synthetic loop, and the difference is not subtle. An agent that spends its day reading files, proposing edits, and running tools feels categorically more responsive when the underlying generation nearly doubles. The lag between asking for a change and watching it appear compresses to the point where the tool stops feeling like a slow correspondent and starts feeling like a pair.

Then I asked it to do something that was not code, and the machine went back to its old pace.

The trick is a smaller model doing the guessing

To understand why the speedup is so uneven, you have to understand where it comes from, because the source of the acceleration is also the source of its limits. The technique is called multi-token prediction, and the underlying idea is speculative decoding, which has been circulating in inference circles for a couple of years but is now wired into Ollama for this model by default.

Here is the shape of it. A large language model normally produces text one token at a time, and each token requires a full pass through the network, which on a big model is expensive. Multi-token prediction bolts a second, much smaller draft model onto the process. That draft model runs ahead and proposes the next several tokens cheaply. The main model then verifies the entire proposed run in a single pass, keeping the tokens it agrees with and discarding the rest. When the draft guesses correctly, you have committed several tokens for roughly the price of one, and the output is identical to what the model would have produced alone. Nothing about the quality changes. You are not trading accuracy for speed. You are trading redundant computation for a gamble that a cheap model can predict what an expensive model was going to say anyway.

The elegance is that when the gamble fails, you lose very little. Ollama tracks how often the draft model’s proposals are accepted and tunes how many tokens to speculate on in real time. When acceptance rates collapse, it quietly falls back to plain one-at-a-time decoding, so speculation never makes generation slower than it would have been without the feature. That auto-tuning is the hard part of the engineering, and it is the reason the thing is safe to leave on.

Code is predictable, which is the whole point

Now the conditional part comes into focus. Speculative decoding only pays off when the draft model guesses right often enough that the saved passes outweigh the wasted ones. And whether it guesses right depends entirely on how predictable your text is.

Code is spectacularly predictable. It is full of closing brackets that follow opening ones, repeated identifiers, boilerplate that appears in the same shape thousands of times, import blocks, type signatures, and the endless small ceremonies of syntax. A tiny draft model can anticipate a great deal of that with high confidence, so acceptance rates stay high and the main model spends most of its time confirming good guesses rather than generating from scratch. This is why Ollama benchmarked the feature on a coding agent in the first place. Coding agents call the model continuously, and the workload is exactly the sort of repetitive, structured text where a cheap predictor thrives.

Prose is the opposite. When I asked the model to rewrite a paragraph, it was navigating a space of word choices that a small draft model has no reliable way to anticipate. Natural language at the sentence level is far less constrained than a function body. The draft proposes, the main model rejects, and the acceptance rate falls low enough that speculation stops helping. The auto-tuner notices, backs off, and you are left generating one token at a time at the model’s baseline speed. The ninety percent was never a property of the model or the chip. It was a property of the text.

The number that travels, and the number that does not

So there are two figures hiding inside the single headline, and it is worth separating them cleanly, because conflating them is how people end up disappointed.

The first figure is the best case: nearly ninety percent faster, on a coding-agent workload, on capable Apple Silicon, with Gemma 4 specifically. That number is honest, it is reproducible, and if your daily use of a local model is agentic coding, you should expect to feel it. On an M5 Max I did feel it, immediately and consistently, for that kind of work.

The second figure is the one that travels less well. For chat, for writing, for summarization, for question answering, for anything where the next token is truly uncertain, the gain shrinks toward nothing. Not because the feature is broken but because there is simply less predictable structure for the draft model to exploit. There is also a hardware dimension worth stating plainly for readers who are not on an M5 Max. Reports from smaller machines put the multi-token gain closer to one and a half times rather than double, and the speculative approach leans on a specialized matrix-multiplication kernel that Ollama contributed to MLX, which favors newer, faster Apple Silicon. Your mileage narrows on both axes at once, the workload and the silicon.

There is a scope limit on top of all that. Gemma 4 is the first model to receive this treatment, and Ollama has said more will follow. If you have standardized on Qwen or DeepSeek or anything else for your local work, the feature does not apply to you today at all, regardless of how much code you write or how fast your Mac is.

When the speedup is worth chasing

None of this is a reason to skip the update. It is a reason to calibrate what you expect from it, which is a different thing entirely.

If you run a coding agent against a local model on Apple Silicon, update tonight, re-pull Gemma 4, and enjoy a real and free improvement to a workflow you already have. The gain is real, the output is unchanged, and there is no downside because the auto-tuner protects you when speculation stops paying. This is close to a pure win for one specific and increasingly common use case.

If your local model is mostly a chat companion, a writing assistant, or a research tool, temper the excitement. You will not see anything like ninety percent, and you may not notice a difference at all. That is not a failure of your setup. It is the predictable result of asking a technique that feeds on repetition to speed up text that has very little of it.

The larger lesson, and the one I keep relearning with local AI, is that a single benchmark number is almost never a promise about your experience. It is a promise about a specific workload on specific hardware with a specific model, and the interesting engineering questions live in the distance between that specificity and whatever you actually do all day. Multi-token prediction is a clever piece of work. It just happens to be clever about code, and the moment you stop coding, it politely steps aside.