llama.cpp adds MTP for the Qwen3.6 family

This is a significant milestone for the local AI ecosystem. The performance jump with these changes is massive and elevates local inference on commodity hardware further.

Special thanks to Aman Gupta for leading this development!

https://github.com/ggml-org/llama.cpp/pull/22673

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀

Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%).

Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

finally faster Qwen3.6 models with MTP support ⚡️

brb updating my Pi & Hermes setup 🤝

@LongOptimist23b @victormustar How many t/s are you getting?

@julien_c I wrote a gist about this with all the flags that I've been using. 35B works super smooth on my Mac M4 Pro 48 GB, and 27B is usable as well. The MTP update definitely helped.

https://gist.github.com/eeshansrivastava89/85797104af34181944bfd1360d69e8af

@victormustar seems shitty and slow try llama-server flags: -ngl 99 -c 262144 -np 1 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --port 8080 --host 0.0.0.0

@victormustar That only works on dense models right? And you have to load a second llm and run that at the same time? No real gain with moe models is that correct?

@UnslothAI @Alibaba_Qwen You have a typo in the MTP Qwen3.6-27B code sample:

Instead of:

-hf unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL \

It should be:

-hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL \

Also it would be amazing to have similar tests/results with AMD GPU ROCm hardware

@ggerganov its cute - but we gotta consider diffusion instead of auto-regression - I need more compute to verify claims - but it's alegedly 4x on base qwen 2.5 - testing on qwen 3.6 now - https://github.com/pengzhangzhi/Open-dLLM/pull/27

@ggerganov performance gains this are what make local ai move from cool demo to daily use tool. i'm building local ai on the iphone side, and the pattern is the same the more reliable local inference gets, the more private workflows become possible

@Alibaba_Qwen Here’s 4-bit Qwen3.6-27B-MTP-GGUF hitting 96.4 tokens/s in Unsloth Studio on an H100.

Try it yourself: https://github.com/unslothai/unsloth

MTP = Multi-Token Prediction.

It's a new speculative decoding feature in llama.cpp for Qwen3.6 models (special MTP GGUF versions). The model predicts multiple tokens at once instead of one-by-one, giving ~1.5-2x faster generation with no quality loss.

Enable with --spec-type draft-mtp. Game changer for local inference.

@ggerganov Its fantastic - thanks a lot to both of you! For me its a game changer with my hardware from 2017 ...

@victormustar @ClementDelangue i was already building llama.cpp from source and using dflash+turboquant. not sure MTP is going to beat that

@_z_I_G_G_Y_ no it works on MoE too, just way less gain: to check N drafts at once you have to load all the experts they touch, so you lose the MoE speed advantage. (from my tests: dense ~+78% & for the A3B ~+10%)

@Chris65536 @HarrisDePercept @UnslothAI @Alibaba_Qwen These are MTP numbers. It's a little over half in either case without it. I can run the 27b at 6bit without it all day but I prefer the shorter feedback. Models are great but they're dumb so I'd rather have more attempts on goal and have them "remember what didn't work".

@UnslothAI @Alibaba_Qwen I'm new to local LLM's, do I need a GPU?

I'm having 64GB RAM and 6GB VRAM with a nvidia card

@victormustar AFAIK you lose a lot of quality with --spec-draft-n-max 2 , from my understanding 3 is widespread considered the bottom, 4 is optimal, and higher than that is pretty much lossless but at the cost of substantially reduced speed gains, but I may be wrong.

@uttertard mhh I dont see how you loose quality: spec decoding is lossless by design. the main model verifies every draft token, so output is identical regardless of n-max. so (at least from my undersanding) n only affects speed, not quality.