I Tested MTP Speculative Decoding on Two Qwen Models — One Was a Trap

You enable speculative decoding, see generation speed jump 67%, and think you’ve cracked fast local inference. Then you check your context window and realize it just collapsed from 80K to 40K tokens.

The speedup is real. So is the cost. And on a 16 GB GPU, that cost might be more than you’re willing to pay.

Why This Matters

Most developers with consumer GPUs hit the same wall: their model runs, but it’s slow. Speculative decoding with Multi-Token Prediction (MTP) promises a fix — the model predicts multiple tokens per forward pass, boosting throughput without changing output quality.

But MTP is not a free lunch. The prediction heads consume VRAM. On a 16 GB card, that VRAM comes directly out of your KV cache budget, which shrinks your context window. If your agent needs 64K context to function, and MTP compresses it to 40K, you’ve broken your workflow to gain speed.

The question isn’t whether MTP works. The question is which model, which quantization, and which settings give you a usable trade-off — and which combination silently breaks your pipeline.

TL;DR

• Qwen 3.6 27B + MTP is a genuine win on 16 GB GPUs — up to 67% faster generation at the cost of context window.
• Qwen 3.6 35B MoE + MTP collapses the context window to 10–15K tokens, making it impractical for real workloads on 16 GB.
• **--spec-draft-n-max 2 with q8 KV cache** is the sweet spot for the 27B model: fastest generation speed.
• q5 KV cache recovers context but degrades quality — test it on your own tasks before committing.
• Standard decoding wins for 35B on 16 GB unless you have 24+ GB VRAM.

What I Tested

I ran Qwen 3.6 27B (dense) and Qwen 3.6 35B (MoE) on an RTX 4080 with 16 GB VRAM, comparing MTP speculative decoding against standard autoregressive decoding. Both models were tested with MTP-enabled GGUF files and their standard counterparts.

The quantizations used were heavily compressed to fit in VRAM: IQ3_XXS for 27B and IQ3_S for 35B. Two KV cache levels were tested — q8 for quality and q5 for context length.

I tracked two context metrics per configuration:

• Avg Ctx — the context size at which the GPU sits at a practical ~14.8 GB usage, leaving room for desktop apps.
• Max Ctx — the absolute maximum context llama.cpp could allocate.

Avg Ctx is the number that matters for daily work. Max Ctx is theoretical.

For broader context on how VRAM limits, memory bandwidth, and runtime scheduling shape LLM inference performance, the LLM performance benchmarks on this site cover the constraint hierarchy and optimization strategies in detail.

Qwen 3.6 27B: MTP Delivers Real Speed

The 27B dense model is where MTP shows its strongest case.

With q8 KV cache and --spec-draft-n-max 2, generation speed jumps from 45 t/s to 75 t/s — a 67% improvement. Prompt ingestion drops from 200 t/s to roughly 150 t/s because MTP requires device-to-host transfers during prefill. The average context window halves from 80K to 40K tokens.

That trade-off is deliberate. You accept a smaller context window for significantly faster generation. For many single-turn or moderate-context tasks, 40K is more than enough.

With q5 KV cache, the story shifts. --spec-draft-n-max 1 gives 57 t/s with 70K average context — a 39% speedup over standard decoding while preserving a much larger working window. At --spec-draft-n-max 3, generation reaches 67 t/s but context drops to 60K.

Here is where the numbers matter most:

Strengths

• 67% generation speedup with q8 KV + MTP max 2
• 39% speedup with q5 KV + MTP max 1, while keeping 70K context
• MTP GGUF files work in both speculative and standard mode — no need to keep two model files

Weaknesses

• Prompt ingestion slows from 200 to ~150 t/s
• Average context drops 50% with q8 KV cache
• q5 KV cache recovers context but introduces noticeable quality degradation

Result

• MTP is genuinely worthwhile for the 27B model on 16 GB. The sweet spot is q8 KV + --spec-draft-n-max 2 for raw speed, or q5 KV + --spec-draft-n-max 1 for balance.

Qwen 3.6 35B MoE: Speed Without a Usable Context Window

The 35B model uses a Mixture-of-Experts architecture — 35B total parameters, but only ~3B active per token. The sparse routing means the MTP head is computationally cheap relative to a full forward pass, so the speed gains should be impressive.

They are. And they’re completely useless.

With q8 KV cache, MTP at --spec-draft-n-max 1 gives 186 t/s — a 27% improvement over standard 146 t/s. But the average context window is 15K tokens. At --spec-draft-n-max 2, it drops to 10K. At max 3 and 4, there is no viable average context at all.

Switching to q5 KV cache doesn’t rescue the situation. --spec-draft-n-max 1 gives 10K average context at 151 t/s. Standard decoding at q5 gives 122 t/s with 120K average context.

Strengths

• 27–29% generation speedup over standard decoding
• MoE architecture keeps MTP head computationally cheap

Weaknesses

• Average context collapses to 10–15K tokens — barely enough for a single conversation turn
• No draft depth beyond 1 keeps any usable context
• q5 KV cache marginally improves context but doesn’t reach practical levels

Result

• MTP is not practical for the 35B MoE model on a 16 GB GPU. Standard decoding at 122–146 t/s with 80–120K context is significantly more useful. If you have 24+ GB VRAM, the context window issue disappears and MTP becomes attractive.

The Hidden Tax: What MTP Actually Costs

The numbers above tell a story that’s worth repeating in plain language. MTP speculative decoding doesn’t add a fixed overhead — it consumes VRAM proportionally to your --spec-draft-n-max setting. Those draft buffers eat directly into your KV cache allocation.

On a 16 GB card, every megabyte matters. The MTP heads add roughly 1–2 GB of overhead. That’s not a lot in absolute terms, but it’s enough to push the KV cache allocator into a tighter bracket, which cascades into a smaller context window.

The concrete failure case is worth highlighting. Consider an agentic workflow that requires 64K context to maintain tool-calling state across multiple steps:

Standard 27B + q8:    80K avg context  →  Agent works fine
MTP 27B + q8 max 2:   40K avg context  →  Agent fails at step 3
MTP 27B + q5 max 1:   70K avg context  →  Agent works, but slower
MTP 35B + q8 max 1:   15K avg context  →  Agent fails immediately

The agent doesn’t gradually degrade. It hits a hard wall at 64K and rejects the model. There is no partial credit for “almost enough context.”

Surprises

Three things stood out from these benchmarks.

The first surprise was how effective MTP is on the 27B dense model. A 67% generation speedup is substantial — most optimizations in local inference land in the 10–20% range. MTP delivers an order-of-magnitude improvement in perceived responsiveness.

The second surprise was how hard the 35B MoE model fails at context window. The MoE architecture is supposed to be the ideal candidate for MTP because the sparse routing keeps the prediction head cheap. The speed numbers prove that — 186 t/s is impressive. But the VRAM math doesn’t work out on 16 GB. The model is simply too large.

The third surprise was the quality drop from q8 to q5 KV cache. The benchmarks show that q5 recovers meaningful context window, but the response quality degradation was significant enough to make q5 unsuitable for my workloads. Speed and context are meaningless if the model starts producing incoherent output.

Practical Ranking

Here is how the configurations rank for a 16 GB GPU (RTX 4080):

1. Best overall: Qwen 3.6 27B + MTP, q8 KV, --spec-draft-n-max 2

• 75 t/s generation speed, 40K average context
• Best raw throughput, acceptable context for most tasks

2. Best balance: Qwen 3.6 27B + MTP, q5 KV, --spec-draft-n-max 1

• 57 t/s generation speed, 70K average context
• Preserves enough context for agentic workflows

3. Best standard option: Qwen 3.6 35B, standard decoding, q8 KV

• 146 t/s generation speed, 80K average context
• No MTP overhead, full context window, still fast

4. Not recommended: Qwen 3.6 35B + MTP on 16 GB

• 186 t/s generation speed, 10–15K average context
• Speed is impressive but context window is too small for real work
• Only viable on 24+ GB VRAM

Key Lessons

The takeaway from these benchmarks is simpler than the numbers suggest.

MTP speculative decoding works best when the model is small enough that the MTP overhead doesn’t break your context budget. For a 16 GB GPU, that means the 27B dense model is the sweet spot. The 35B MoE model, despite being architecturally optimized for MTP, is too large for consumer hardware to handle both the model weights and the prediction buffers.

The --spec-draft-n-max setting is your primary tuning knob. Higher values increase VRAM pressure without proportional speed gains — at max 4, you're spending roughly the same extra VRAM as max 2 but generation speed doesn't keep pace. Start at 2, test your context window, and adjust from there.

And always test q5 KV cache on your own tasks. The quality drop is real, and it’s workload-dependent. What’s acceptable for casual chat may be unacceptable for code generation.

👉 Qwen 3.6 27B and 35B MTP vs Standard on 16GB GPU