Supercharge Your Local LLM: Unlocking MTP in llama.cpp

Stop waiting for tokens. Learn how to activate Multi-Token Prediction and accelerate your local AI agent without extra hardware.

Source: Image by Shirley Heyn on Shirley Heyn

Table of Contents

1. Introduction
2. Demystifying MTP: Do I Need Two Models?
3. Update llama.cpp
4. Get an MTP-Enabled GGUF
5. Update Your Execution Script (.bat)
6. When Does MTP Shine?
7. Conclusion
8. References

💥 Master Any Skills in 3 Months

• 📚 Up to 50% OFF Premium Courses ⏰ Limited-Time Offer [👉 **Enroll Now & Start Learning](https://trk.udemy.com/zz4NBO)***

Introduction

If you followed my previous guide on how to run a local AI coding agent using llama.cpp and Qwen Code, you already have a fully private, autonomous assistant living on your machine. But what if we could make it generate code significantly faster without buying a new GPU?

Enter MTP (Multi-Token Prediction). Recently merged into the main branch of llama.cpp, this feature allows compatible models to guess multiple words at once, drastically increasing your tokens-per-second (t/s) during code generation.

Let’s break down how to activate it, what you need, and demystify how it works.

Demystifying MTP: Do I Need Two Models?

The short answer is No. Typically, to speed up an LLM using “Speculative Decoding,” you need to load two separate models into your RAM: a massive, smart model (which is slow) and a tiny, dumb model (which is fast). The tiny model guesses the next few words, and the big model verifies them. If the guess is right, you get a burst of speed. The problem? Managing two models is a nightmare for standard PC hardware.

MTP solves this elegantly. Models trained with MTP (like the Qwen 3.5 and 3.6 families) have a tiny “draft” sub-model built directly into their weights. You only download and load one file, but you get the speed benefits of two.

Update llama.cpp

MTP support is a very recent addition. Your older version of llama.cpp won't recognize the new commands.

1. Head to the official llama.cpp releases page.
2. Download the absolute latest compiled release for your hardware (e.g., Windows CUDA and CUDA DLLs).
3. Extract and overwrite your existing executable files.

Step 2: Get an MTP-Enabled GGUF

Here is the catch: when community members quantize models to the .gguf format, they often strip out the MTP sub-model to make the file slightly smaller.

To use MTP, you must download a model explicitly packaged with it.

• Go to Hugging Face and search for [unsloth/Qwen3.5-4B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.5-4B-MTP-GGUF). The Unsloth team regularly uploads versions with the MTP layers intact.
• Example: Download the Qwen3.5-4B-MTP-GGUF (or a similar size that fits your VRAM). You will notice it is slightly larger than the standard version—that's the hidden sub-model doing its job.

Update Your Execution Script (.bat)

Remember the .bat file we created in the last guide to boot up our local API server? We only need to add a couple of flags to awaken the MTP feature.

Open your .bat script and add the following parameters:

.\llama-b8873-bin-win-cuda-13.1-x64\llama-server.exe ^
  -m gguf\Qwen3.5-4B\Qwen3.5-4B-UD-Q4_K_XL.gguf ^
  -c 262144 ^
  --spec-type draft mtp ^
  --spec-draft-n-max 3 ^
  -mg 1 ^
  -sm none ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --image-min-tokens 1024 ^
  --reasoning on ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.00 ^
  --presence-penalty 0.0 ^
  --repeat-penalty 1.0 ^
  --dry-multiplier 0.1 ^
  --dry-base 1.05 ^
  --dry-allowed-length 12 ^
  --dry-penalty-last-n 128 ^
  -a Qwen3.5-4B

What changed?

• --spec-type draft mtp: This tells llama.cpp to stop generating token by token and start using the embedded MTP sub-model for speculative decoding. (Note: in older builds, this was -mtp or -spec mtp, but the syntax has been standardized to --draft mtp).
• --spec-draft-n-max 3: This specifies how many tokens the sub-model should try to predict ahead of time. Starting with 3 is a sweet spot for stability and speed.

When Does MTP Shine?

Manage your expectations: MTP is not a magic wand for every scenario.

• Where it wins (Coding & Reasoning): You will notice a significant speed boost when the AI is writing code blocks, CSS, or JSON structures. Because syntax is highly predictable, the small MTP sub-model guesses correctly almost every time, letting the main model fly through the generation.
• Where it stalls (Huge Context Prefill): MTP accelerates the generation of new tokens, not the reading of your prompt. If you feed it a massive codebase, the initial “thinking” phase might actually be slightly slower as it sets up the MTP context.

Conclusion

By simply updating your llama.cpp binary and swapping your .gguf file for an MTP-enabled version, you can squeeze out extra performance for free.

Fire up Qwen Code or OpenCode, ask them to build a new feature, and watch the terminal fly.

References

• unsloth/Qwen3.5–4B-MTP-GGUF · Hugging Face. (2026, May 18). Huggingface.Co. https://huggingface.co/unsloth/Qwen3.5-4B-MTP-GGUF
• Nichonauta. (17 de mayo de 2026). MTP 🚀 ADIÓS A LOS SERVICIOS DE PAGO para Siempre. YouTube. https://www.youtube.com/watch?v=Gq2UI6xjwMI

Thank you for being a part of the community

Before you go:

👉 Be sure to clap and follow the writer ️👏️️

👉 Follow us: **Linkedin| [Medium](https://medium.com/codetodeploy)**

👉 CodeToDeploy Tech Community is live on Discord — **Join now!**

Disclosure: This post includes affiliate and partnership links.‌