Running 26B and 35B LLMs at Full Speed on €990 of Used Hardware — No Cloud Required

How a secondhand gaming PC keeps pace with a single RTX 3090 for local AI inference — and what I learned measuring it honestly, including the speculative-decoding “speedup” that turned out to be a measurement mirage.

The €990 machine: an RTX 4070 and an RTX 2070 SUPER sharing a case nobody would mistake for a workstation

On June 3, 2026, Google added a 12B model to the Gemma 4 family it launched back in April. The pitch: frontier-ish performance that “runs on laptops” with 16 GB of VRAM, benchmarks nearly matching the larger 26B sibling. The tech press called it a milestone for local AI.

We were already running the whole family. The 12B at 50 tokens per second. The 26B at 83. And a 35B mixture-of-experts model from Alibaba at 73. All on the same machine — assembled from secondhand parts for €990.

That’s the story worth telling. Not which model dropped this week, but the fact that the floor for serious local inference has quietly fallen through. You no longer need a workstation. You need two used gaming cards, and the patience to measure what they’re actually doing — because, as it turns out, one of the “obvious” speed tricks makes things slower on this kind of setup, and you only catch it if you test honestly.

The Machine

Nothing here is exotic, and nothing is new. Every part is sitting on a secondhand marketplace right now, today.

Every part bought used. Two gaming GPUs, 20 GB of combined VRAM, ~€990 all in

The trick is the pairing. A 2070 SUPER next to a 4070 gives you 20 GB of combined VRAM — not from one expensive card, but from two that nobody wanted anymore, bought at the bottom of their depreciation curve. The 4070 brings modern Ada throughput; the 2070 SUPER brings cheap capacity. Together they punch well above what either does alone.

The OS is antiX Linux — SysV init, no systemd. Lean, predictable, and it stays out of the way. On a machine where every gigabyte of RAM counts, that minimalism is a feature, not nostalgia.

llama.cpp, Not Ollama

Ollama is the comfortable choice, and for good reason. It fetches models, serves a clean API, and works the moment you install it. For a first experiment it’s excellent.

But comfort has a cost: it hides the exact controls you need once you start pushing the hardware. llama.cpp exposes every one of them — per-GPU layer distribution (--tensor-split), flash attention, expert offloading for MoE models, and speculative decoding via Multi-Token Prediction (merged to mainline in May 2026, PR #22673).

Here’s the picture across all three models — and one note on method that matters more than it sounds. Every number here is measured on a sustained 2000-token generation, not a short burst. That distinction turned out to be the whole ballgame, as you’ll see in the MTP section.

Sustained throughput across backends. llama.cpp opens a 35–38% gap on the MoE models; for Qwen, the IQ4_XS quantization adds another jump to 73 tok/s

The same numbers in full. Ollama runs its default Q4_K_M; the llama.cpp gains are measured at the same quantization, with Qwen’s best-config IQ4_XS shown separately

llama.cpp beats Ollama by 35–38% on the MoE models, measured at the same quantization (both running Q4_K_M, the default Ollama pulls). For Qwen there’s a second, separate gain on top: switching to the IQ4_XS quantization — same quality, fewer operations per token on this architecture — takes it from 61.9 to 73 tok/s. Two independent levers, kept apart on purpose: backend first, quantization second. (More on why I’m so careful to isolate them in the MTP section — that’s where mixing variables nearly cost me a wrong conclusion.)

One honest caveat: Ollama v0.30.5 is no longer naive about MoE models. It runs Qwen3.6 35B-A3B — 21 GB, a full gigabyte over total VRAM — at a respectable 46 tok/s, clearly doing some automatic expert scheduling under the hood. The gap with llama.cpp is real, but it isn’t the chasm older comparisons suggested. If you never leave Ollama, you’re not stranded. You’re just leaving a third of your throughput on the table.

And that gap is the whole point. Where llama.cpp earns its complexity is the control surface: -ncmoe lets you pick exactly how many MoE expert layers spill to CPU RAM (more on this below); --tensor-split sets per-GPU layer ratios at single-percent granularity; --no-mmap stops cache thrashing on the big models.

Model 1: Gemma 4 26B — Fits Cleanly, Goes Fast

Gemma 4 26B-A4B is a mixture-of-experts model: 26B total parameters, but only ~4B active per token. The Q4_K_M quantization lands at 16.9 GB — a tight but clean fit across the two cards.

With --split-mode layer --tensor-split 0.62,0.38 --flash-attn on, the layers distribute proportionally to each GPU's VRAM:

RTX 4070   (12 GB): 62% of layers
RTX 2070S   (8 GB): 38% of layers

The result: 82.6 tok/s, sustained and stable across runs and generation lengths. That’s comfortably in RTX 3090 territory — a card that still costs €800–1,000 by itself used, gives you 24 GB of unified VRAM, and offers no upgrade path. This build matches it for less money and leaves both slots open.

Model 2: Qwen3.6 35B-A3B — The One That Doesn’t Fit

Qwen3.6 35B-A3B is a different animal. Released by Alibaba in April 2026, it’s an MoE model with 35B total parameters but only ~3B active per forward pass. The Q4_K_M quantization is 21.2 GB — 1.5 GB more than the VRAM you have. It doesn’t fit. That’s where it gets interesting.

llama.cpp’s -ncmoe flag offloads the first N layers of MoE experts to system RAM, relieving GPU pressure at the cost of some CPU bandwidth. Finding the right N was iterative:

Tuning expert offload for the model that doesn’t fit. Splitting purely by VRAM capacity OOMs the smaller card — the 70/30 split leaves it room to breathe

The lesson the docs won’t tell you directly: tensor-split ratios based on raw VRAM capacity aren’t enough. The smaller GPU — the 8 GB 2070 SUPER — also needs headroom for the KV cache and compute buffers, roughly 1–1.5 GB. Split purely by capacity and it OOMs on the compute buffers before it finishes loading. Shift to 70/30 and you protect it.

That gets the Q4_K_M build to 61.9 tok/s. But there’s a better quantization for this model: IQ4_XS packs the same quality into fewer per-token operations on this architecture, fits a little more on the GPUs, and lands at 73 tok/s sustained — the fastest config I found for Qwen, and the one I’d recommend. Worth noting the win comes from the quantization, not from any speculative-decoding trick. (Hold that thought.)

Qwen3.6 is a refinement of Qwen3.5 — same MoE architecture, better training data — and it’s particularly strong for agentic and coding work. Native 262K context, reasoning mode built in, and in independent benchmarks it consistently outperforms models well above its active-parameter weight class.

Multi-Token Prediction: A Lesson in Not Fooling Yourself

MTP is a form of speculative decoding: the model predicts several tokens ahead at once, and the main forward pass verifies them. When the predictions are right, you get multiple tokens for the price of one pass. No quality loss either way. On paper, free speed. The internet is full of “+18% with MTP!” posts. I wanted that 18%.

My first measurements seemed to deliver it — on short prompts, Gemma 4 26B with an external drafter nudged from ~83 to ~86 tok/s, and the Qwen MTP build came in at ~66 against a ~60 baseline. I nearly wrote both up as wins. Two things stopped me, and both are worth more than the result itself.

First: short prompts lie. I re-ran on a sustained 2000-token generation, and the Gemma external-drafter setup didn’t gain anything — it collapsed from 82.6 to 48.5 tok/s, a 41% loss. On a dual-GPU box the drafter and the main model sit on different cards, so every draft-verify cycle crosses the PCIe bus. On a short prompt that overhead is invisible; over a long generation the synchronization bubbles compound token after token until they swallow the speculative gains whole, then keep eating. The fork works exactly as advertised — the hardware topology is what defeats it.

Second, and more embarrassing: my Qwen “+7%” wasn’t from MTP at all. The MTP build used a different quantization (IQ4_XS) and a different offload setting (-ncmoe 10 vs 15) than the baseline. Two variables changed at once. So I isolated them:

The measurement that nearly fooled me. Once quantization and offload are held constant, adding MTP doesn’t speed Qwen up — it slows it down 9%. The apparent gain was the quantization all along.

Held at the same quantization and offload, MTP doesn’t speed Qwen up — it slows it down 9% (73.0 → 66.4 tok/s). The entire apparent gain came from IQ4_XS being a faster quantization and from -ncmoe 10 putting 2 GB more on the GPU. The MTP heads were a tax the whole time; the quantization change just more than paid for it. Embed the heads or run them as a separate drafter — either way, the draft-verify cycle costs PCIe bandwidth this build doesn't have to spare.

So here’s the honest conclusion, and it’s cleaner than the one I set out to write: on a consumer dual-GPU NVIDIA box, speculative decoding via MTP doesn’t pay off. Not with an external drafter, not with embedded heads. The spectacular numbers you’ll see quoted come from unified-memory Macs (one shared memory bus, no PCIe hop) or from datacenter cards linked by NVLink (hundreds of GB/s between GPUs). Neither describes a pair of secondhand gaming cards talking over PCIe. The mechanism is sound; the interconnect is the bottleneck.

Practical takeaway: skip MTP on this kind of hardware. For Qwen3.6, the real win is the IQ4_XS quantization at 73 tok/s — no speculation required. For Gemma 4, run the model straight: 82.6 tok/s sustained beats anything the drafter produces.

And the meta-lesson, the one I’d hand to anyone benchmarking inference: change one variable at a time, and measure on long generations. I came within one edit of publishing a “+7% from MTP” that was really “+18% from a quantization swap minus 9% from MTP.” Short prompts and tangled variables will tell you whatever you want to hear.

Power & Efficiency — The Surprise Nobody Talks About

Two used GPUs sounds like a lot of power. It isn’t — and the reason is the same MoE architecture that makes these models fast. I measured sustained decode at the wall, cross-checked with per-GPU draw in nvtop:

Measured at the wall and cross-checked in nvtop. The 12B dense model draws nearly twice the GPU power of the 35B

The MoE efficiency cliff, in one chart. A 35-billion-parameter MoE model is more power-efficient per token than a dense model a third its size.

The headline is genuinely counterintuitive: the 12B dense model draws more power than the 35B MoE. 204 W versus 108 W on the GPUs — for a model with a fraction of the parameters. With MoE, only ~3B parameters activate per token, so the GPU does far less arithmetic even though the model is enormous on disk. Dense models light up every parameter every token; MoE models don’t. The result is that the bigger model is the cooler, cheaper one to run.

Put it in money terms. At 82.6 tok/s and €0.25/kWh (a reasonable Italian rate), the 26B MoE generates roughly 1 million tokens for about €0.22 — call it 3.4 hours of sustained decode at ~257 W system draw. That’s not a typo. A million tokens, locally, privately, for the price of a coffee.

Why This Matters Now

We’re in a strange moment for compute. GPU and HBM prices keep climbing — not because gamers are buying, but because datacenter orders are swallowing entire production runs. An RTX 4090 that launched at €1,200 now goes for €1,800 used. High-bandwidth memory is effectively rationed.

Against that backdrop, multi-GPU secondhand builds are quietly becoming one of the most cost-effective routes to serious local inference. The reasons stack up:

Privacy. Every token generated locally is a token that never crosses an API. For coding assistants, document analysis, internal tooling — that’s the difference between using a model and feeding your codebase to someone else’s.

Latency. At 73–83 tok/s sustained on the big models, interaction feels real-time. No round-trip to a server farm, no rate limits, no quiet degradation during peak hours.

Cost — and power — at scale. €0.22 per million tokens, on hardware that already paid for itself. The MoE efficiency means you’re not running a space heater to get there.

Upgradeability. Swap the 4070 for a 4090 later. Turn the 2070 SUPER into a second 4070. The RAM, the NVMe, the case all stay. VRAM grows incrementally instead of all at once.

The Bigger Picture

None of these models are toys. Gemma 4 26B and Qwen3.6 35B-A3B are competitive with GPT-3.5-class performance on most practical benchmarks, and Qwen punches far above its active-parameter count on coding and reasoning.

Running them locally, at full speed, privately, for cents per million tokens, on hardware that costs less than a single mid-range GPU did two years ago — that’s the actual state of things in mid-2026. The datacenter isn’t the only place serious inference happens anymore.

The €990 build isn’t a compromise. It’s a choice.

Next: how much do you actually give up versus a single RTX 3090 or an A6000? I put all three on the same bench and measured throughput, power, and cost per token — real numbers, not spec-sheet estimates. Coming in the follow-up.

Test bench. AMD Ryzen 7 3700X (8C/16T) · 32 GB DDR4–3000 · RTX 4070 12 GB + RTX 2070 SUPER 8 GB (20 GB combined) · 2 TB NVMe · antiX Linux (SysV init)

Inference. llama.cpp build 9307 (549b9d843), custom CUDA build (sm_75 + sm_89), GCC 14.3.0 · Ollama v0.30.5 for the baseline, running its default Q4_K_M

MTP. Tested via mainline --spec-type draft-mtp and the am17an/llama.cpp mtp-clean fork. Not recommended on this dual-GPU topology — see the MTP section for why.

Models. Gemma 4 12B (UD-Q4_K_XL) · Gemma 4 26B-A4B-it Q4_K_M (unsloth) · Qwen3.6 35B-A3B IQ4_XS (reference config), Q4_K_M (Infatoshi) for the matched-quantization backend comparison

Method. All figures are sustained throughput over a 2000-token generation, not short bursts. Power measured at the wall and cross-checked against per-GPU draw in nvtop. Energy cost at €0.25/kWh.

Sources

• Qwen3.6–35B-A3B on Hugging Face
• ironbcc/gemma-4–26B-A4B-it-MTP-GGUF
• am17an/llama.cpp — mtp-clean branch
• localweights/Qwen3.6–35B-A3B-MTP-IQ4_XS-GGUF
• llama.cpp PR #22673 — MTP support

AI assistance: the benchmarks were run by me on the hardware described above. I used an AI coding agent to help execute the test runs and parse results, and an AI assistant to help draft and edit this article. Every number was measured on real hardware, and every claim was cross-checked against primary sources and re-tested when it didn’t hold up — the MTP section is what that process looks like when it catches something.