The update follows the integration of MTP in llama.cpp.
In HF GGUF section of models, we are emphasizing MTP heads with its own sign 𝗠𝗧𝗣
llama.cpp adds MTP for the Qwen3.6 family
This is a significant milestone for the local AI ecosystem. The performance jump with these changes is massive and elevates local inference on commodity hardware further.
Special thanks to Aman Gupta for leading this development!
https://github.com/ggml-org/llama.cpp/pull/22673
No Digg Deeper questions have been answered for this story yet.
In HF GGUF section of models, we are emphasizing MTP heads with its own sign 𝗠𝗧𝗣
llama.cpp adds MTP for the Qwen3.6 family
This is a significant milestone for the local AI ecosystem. The performance jump with these changes is massive and elevates local inference on commodity hardware further.
Special thanks to Aman Gupta for leading this development!
https://github.com/ggml-org/llama.cpp/pull/22673
example: https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF
@mishig25 I don't understand what's the point of making 2 bit models; the loss of quality is so high that it isn't worth it.
@mishig25 @ggerganov One of my suggestions is to add best parameters config for each model not sure if it is just part of metadata, like what’s the best n_speculate for each
@mishig25 excellent initiative this reduces time spent searching for the mtp
@mishig25 @mervenoyann Very nice!
@mishig25 Why can’t MTP be offloaded?
@mishig25 ty
@mishig25 I don't understand
explain to me like I'm 5
@FacoDeZurdos @mishig25 Compare it against smaller model at higher quant. It as well may be.
@mishig25 Thank you this is great
