Why I Stopped Using Gemma 4 and Switched to Qwen 3.6

Why Qwen’s new Mixture of Experts model is the first open-source LLM that actually handles agentic coding workflows

It was Tuesday night. I was tired. I had been watching Gemma 4 fail at the same task for five minutes straight. The task was simple: read a CSV of customer transaction data, add a column that flagged unusual patterns, run the transformation, save the result. A junior analyst could have written this in ten minutes.

If you cant read the article further because of paywall then please click here

What Gemma produced was something else entirely. It started reasonable, then printed “user asked question, now answer…………… ” back at itself, as if it had confused its own prompt template with the actual instruction. Then it produced a paragraph of gibberish. Then it started a code block, abandoned it halfway, started another one, abandoned that too. No file was ever created. No transformation ever ran. It just kept generating. Five minutes in, it was still going. Still producing nothing. Still stuck in some loop it could not break out of.

I stopped it. I closed OpenCode, opened Claude, and Claude fixed the task in one shot. I sat there feeling stupid for having spent the evening trying to make an open-source model work. Two weeks later, Qwen 3.6 landed on my feed. And this is where the story gets interesting.

The part I did not want to admit

I have been writing about open-source AI for two years. I run a publication called Towards Deep Learning, and thousands of people read what I say about which models are worth their time. The honest truth, until last week, was this: for serious work I used Claude. For everything I wrote publicly about, I used open-source models. The gap between what I recommended and what I actually ran was uncomfortable. Every time a new open model dropped, I got excited, tested it, and quietly went back to Claude.

Gemma 4 was the worst version of this pattern. Google had done everything right on the packaging side. The benchmarks were beautiful. The documentation was clean. I genuinely wanted it to be the open-source model that finally worked for me, and it was my favourite model for about a week. Then I started using it for agentic coding in OpenCode, and the cracks showed up fast.

What Gemma actually did

The failure mode was specific. For one-shot prompts it was fine: ask it a question in a chat interface, it gave a reasonable answer, looked smart. But the moment you wrapped it in an agent loop, something broke. It would start confidently, then lose track of what it was doing. The “user asked question, now answer……. ” leak I hit was not a one-off. It happened repeatedly. The model would get confused about whether it was reading a prompt, generating a response, or planning a tool call. The three got tangled up. Then the output would spiral into paragraphs that looked like text but meant nothing. And it would not stop on its own. You had to kill it manually. Five minutes, ten minutes, fifteen, the token counter ticking up and nothing useful coming out the other end.

[embed]Eval bug: Infinite repetition loop in llama-server with peg-gemma4 parser during tool calls · Issue… When using the newly implemented Gemma 4 (peg-gemma4) chat format in llama-server, the model enters an infinite…github.com

At first I thought it was my setup. Wrong sampling parameters. Bad quantisation. Something on my end. It was not. When I went looking online, the pattern was everywhere. There is an open GitHub issue on OpenCode literally titled “gemma-4–26b and gemma-4–31b opencode interaction issues leading to tool loops/failures.” Developers running the latest patched engines, the latest tokenizer fixes, every recommended setting, all reporting the same thing: tool call failures and command repetition that made the model unusable for agent work. On the Hugging Face discussion page for the official Gemma 4 release, users reported the model getting stuck in what one person called “infinite loops,” splitting words down to individual letters, overthinking simple tasks until it unraveled into nonsense. Same behaviour across image inputs, text inputs, different prompts, different hardware.

This was not a me problem. This was a Gemma problem. The benchmarks they had shown off were mostly one-shot: single question, single answer, grade the answer. Gemma 4 was optimised for that. But agent work is not that. Agent work is twenty turns of tool calls, state management, and recovery from errors. The benchmarks did not measure it. Real work did. I kept trying to make it work anyway with different sampling parameters, different prompts, different quantisation levels. Nothing really fixed the core issue. Eventually I accepted what was happening and went back to Claude for anything that mattered. And that was the state of things when Qwen released Qwen 3.6–35B-A3B.

The number that made me sit up

I almost scrolled past the Qwen post. Another model, another set of internal benchmarks, I was fatigued. What stopped me was one specific number.

MCPMark. Qwen scored 37.0. Gemma 4 scored 18.1. MCPMark measures exactly the thing that had been killing me: tool calls in agent loops, the part every other benchmark ignores. And the gap was not incremental. It was more than double. I stared at that number for a while, because if it was real, it meant someone had finally benchmarked the thing that actually matters for how I use these models. Then I read the architecture page.

Why a 3B model beats a 31B model

Here is the part that broke my brain a little. Qwen 3.6–35B-A3B has 35 billion total parameters, but only 3 billion of them are active for any given token you send it. Gemma 4–31B uses all 31 billion parameters every single time. Smaller active model. Bigger wins.

Benchmark results taken from original blog

The architecture is called Mixture of Experts. Think of it like a hospital. Gemma 4 is a hospital where every doctor, nurse, and technician shows up to see every patient. If you walk in with a sprained ankle, the cardiologist, the neurosurgeon, and the radiologist all crowd into the room. Thorough, but slow, expensive, and overkill. Qwen 3.6 is a hospital with 256 specialists and a really smart triage nurse. You walk in with a sprained ankle, the nurse sends you to three specialists who actually know ankles. The cardiologist stays in her office. You get seen faster and the bill is smaller.

The catch is that the triage nurse has to be good. If she sends the ankle patient to the dermatologist, everything falls apart. Training a good router is hard. It is the part that took the Qwen team three generations to nail. They nailed it.

The benchmarks that actually matter

On SWE-bench Verified, the standard test for real software engineering, Qwen scores 73.4. Gemma scores 52.0. Twenty one point gap, smaller active model, winning. On Terminal-Bench 2.0, agent coding in a terminal, Qwen scores 51.5 to Gemma’s 42.9. On NL2Repo, which tests whether the model can build an entire code repository from a natural language description, Qwen scores 29.4 to Gemma’s 15.5. Almost double.

Every coding benchmark in the chart goes to Qwen. By margins that are not close. And the MCPMark number I started with is still the one that haunts me: 37 versus 18. If you are building anything with tools and agents, that is the number to remember.

The moment I actually tried it

I downloaded Qwen 3.6–35B-A3B that evening. The Unsloth team already had GGUF builds up, llama.cpp ran it without complaints, and I plugged it into OpenCode. I gave it the same data task that had destroyed Gemma. It read the CSV. It understood the schema. It wrote the transformation. It ran the tests. It caught its own mistake on the first run and corrected it on the second. No loops. No gibberish. No confused prompts bleeding into the output. Just the work, done.

I tried a harder task: parse a messy API response, filter for errors, group by category, flag anomalies. This was a real task from my week, one I had originally solved with Claude. Qwen handled it. Slower than Claude, less polished in the final code, but it worked. And the data never left my machine. That was the moment the story changed.

Why this matters beyond one model

For two years I have been telling people that open-source AI is catching up. Every time I said it, I was partly lying. Catching up meant “getting closer but still not good enough for serious work.” This release is the first one where I am not lying anymore. A 3 billion active parameter model, free to download, free to fine-tune, free to ship in a commercial product, running on a 16GB Mac Mini, genuinely usable for real agent workflows. That sentence would have been science fiction a year ago.

The broader point is not about Qwen specifically. It is about what sparse Mixture of Experts architectures now make possible at the open tier. Six months from now there will be a better model. It will probably also be MoE. It will probably be even smaller active parameters and even more capable. The trajectory has flipped. Dense models are the old pattern. Sparse is where the capability lives now.

The caveats I still believe in

Two honest ones before you swap your stack. First, the Qwen chart is Qwen’s own benchmarks on some metrics. The SWE-bench numbers are standard and you can trust them, but some of the internal benchmarks you should verify on your own work before getting excited. Second, Gemma 4 is not bad at everything. On HumanEval specifically, a single-question coding test, Gemma still ranks first in community testing at 100 percent and Qwen sits at 93. If your work is structured like that benchmark, Gemma might still be fine. My work is not structured like that, which is why Gemma failed me. Yours might be different.

What I actually do now

Qwen 3.6–35B-A3B is my default for any coding task where the data should not leave my laptop. Internal scripts, sensitive client work, anything touching information that should stay behind my own firewall, all of it runs local now. Claude is still my reach-for tool when reasoning depth matters more than data locality: long-horizon problems, ambiguous refactoring, anything that needs the extra edge. The closed flagships are still the closed flagships. Gemma 4 is not installed anymore.

If you have been waiting for the right moment to take open-source AI seriously, this is it. Not next year. Not the release after this one. Right now, this week, this model. Go download it. See for yourself.

I think you will feel the same thing I did when I ran that first transformation task and it just worked. Relief. Then excitement. Then the slow realisation that something has genuinely changed.

I write about AI, machine learning, and the tools that actually work. No hype. No fluff. If this was useful, follow me for more. You can also check out thinkidiot.com where I build interactive playgrounds that make complex AI concepts visual and intuitive.