AI Judge changed title after evaluation, original title: "Prime Intellect's Elie Bakouch sparks debate over speculative decoding, with Kalomaze clarifying that draft verification prevents quality degradation"
An AI researcher is probing why speculative decoding once faced resistance, now that DFlash's block-diffusion drafter and DeepSeek's DSpark hybrid have delivered substantial speedups on models like Qwen3-8B while keeping output identical to standard decoding.
@teortaxesTex @antirez @_xjdr @norpadon It's already deployed and working well I'm impressed
It's been very lame how the industry has been failing to adopt good speculative decoding as the baseline. Just like Whale forced everyone onto MTP, now they may succeed with semi-AR drafting. @zephyr_z9 @antirez @_xjdr @norpadon does this look less BS than the previous one?
DFlash reports up to 6x lossless gains versus autoregressive baselines and roughly 2.5x better than prior EAGLE-3 results, with further 3-15x throughput notes on NVIDIA Blackwell and TPU setups via vLLM and SGLang.
Sampled discussion points to past performance degradation concerns with sequential drafters, yet no single origin for the broader doubt is pinned down beyond references to DFlash and related work shifting one critic's stance.
Users reacted with sarcastic disapproval to western inference providers boasting about efficiency gains, in response to DeepSeek open-sourcing DSpark for speculative decoding.
No Digg Deeper questions have been answered for this story yet.
No, they'll just increase the batch size, have the same speed, and drive margins from 90% to 95%. You're welcome
will this finally fix the shitty throughput on every western provider for the love of god
thank you deepseek
@eliebakouch @zephyr_z9 @kalomaze @antirez @_xjdr @norpadon I think Zephyr is wrong tbh There are diffusion-based specdec (sometimes branded as "diffusion LMs", which I hate) which actually degrade output quality, but for MTP we can have guarantees of pure throughput increase (under some batch size etc) models can be bad for other reasons
@zephyr_z9 @kalomaze @teortaxesTex @antirez @_xjdr @norpadon i don't get your point, this doesn't impact model quality if you do it properly, and doing inference optimization is independent of model quality
@eliebakouch @teortaxesTex @antirez @_xjdr @norpadon Mostly the performance degradation
@zephyr_z9 @teortaxesTex @antirez @_xjdr @norpadon what was the reason for hating speculative decoding? 👀
@zephyr_z9 @kalomaze @teortaxesTex @antirez @_xjdr @norpadon i don't get your point, this doesn't impact model quality if you do it properly, and doing inference optimization is independent of model quality
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon step3.5, longcat flash, mimo v2 flash, Gemini - meh models GLM 5.2 using it is the real deal
speculative decoding is used in production since a while (i remember an old jeff dean talk mentioning it was a big deal), glm5.2 100% used it, there is a whole part in the blog about how they do it in a smart way with index and KV sharing
you also have step3.5, longcat flash, mimo v2 flash and basically every open model trained are trained with MTP for this reason
only exception is Kimi (they might have not just open source it btw) but composer 2 for instance is explicit about adding MTP head for inference efficiency
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon I think Whale adopted mtp with V4 and swapped it with Dspark after 2 weeks I don't think V3 series ever used it Does GLM really use it?? Gemini is too stupid My experience with mtp has been really bad Stepfun used it with 3.5 and the output quality was meh
@zephyr_z9 @teortaxesTex @antirez @_xjdr @norpadon i have a question do all of u use dark mode or smth bc the pfps imply this
@menhguin @teortaxesTex @antirez @_xjdr @norpadon It's easy to read in dark mode especially if u are doing it for hours
@zephyr_z9 @kalomaze @teortaxesTex @antirez @_xjdr @norpadon it does not if you do it properly, it is mathematically lossless, once again you can trade efficiency for quality by relaxing the threshold, but this is not needed and you see gain without that
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon It does impact model quality I would recommend you to try running with & without MTP and judge the quality
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon Whale's method handles verification in a much better manner If mtp 3/5 is used then thruput degrades due to verification overhead
@zephyr_z9 @kalomaze @teortaxesTex @antirez @_xjdr @norpadon i agree! was talking about your previous post
> I have been a speculative decoding hater for a long time
Whale's method improves draft quality and manages verification in a much better manner Pair this up with an SRAM based accelerator like Groq then we are off to the races The demand for SRAM based accelerators is growing in China and I think the Whale will try to procure some now
@zephyr_z9 @eliebakouch @teortaxesTex @antirez @_xjdr @norpadon ? drafts are only accepted if they match a true sample
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon I think Whale adopted mtp with V4 and swapped it with Dspark after 2 weeks I don't think V3 series ever used it Does GLM really use it?? Gemini is too stupid My experience with mtp has been really bad Stepfun used it with 3.5 and the output quality was meh
you said that you hate speculative decoding so i was wandering why since my understanding is that it was already heavily used in production by deepseek, GLM and al with MTP (and other US lab like google ect..). we both agree that deepseek new method seems better (see my post above!) and this was not my question but nw
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon step3.5, longcat flash, mimo v2 flash, Gemini - meh models GLM 5.2 using it is the real deal
speculative decoding is used in production since a while (i remember an old jeff dean talk mentioning it was a big deal), glm5.2 100% used it, there is a whole part in the blog about how they do it in a smart way with index and KV sharing
you also have step3.5, longcat flash, mimo v2 flash and basically every open model trained are trained with MTP for this reason
only exception is Kimi (they might have not just open source it btw) but composer 2 for instance is explicit about adding MTP head for inference efficiency
@zephyr_z9 @eliebakouch @teortaxesTex @antirez @_xjdr @norpadon ? drafts are only accepted if they match a true sample
@eliebakouch @teortaxesTex @antirez @_xjdr @norpadon Mostly the performance degradation
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon Maybe I was too stupid to implement it well lol
@zephyr_z9 @kalomaze @teortaxesTex @antirez @_xjdr @norpadon it does not if you do it properly, it is mathematically lossless, once again you can trade efficiency for quality by relaxing the threshold, but this is not needed and you see gain without that
@zephyr_z9 @teortaxesTex @antirez @_xjdr @norpadon what was the reason for hating speculative decoding? 👀
@eliebakouch @teortaxesTex @antirez @_xjdr @norpadon Yes Very Impressive work I have been a speculative decoding hater for a long time Dflash and this from whale have changed my mind
@eliebakouch @zephyr_z9 @antirez @_xjdr @norpadon it's a reference here (about JetSpec)
@teortaxesTex Did you read the paper? It looks strictly bullshit
Wtf is “parallel autoregressive head”???
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon This is better than mtp
@zephyr_z9 @kalomaze @teortaxesTex @antirez @_xjdr @norpadon ok but speculative decoding with MTP was already used by chinese lab in production since a while
@teortaxesTex @zephyr_z9 @kalomaze @antirez @_xjdr @norpadon for V3 i agree this is not sure (tho i think there is a very high chance they use it), for the rest, they do optimization in the MTP layer so that this layer is cheap at inference and/or only do it in mid training there is no doubt it's for inference mainly
@eliebakouch @zephyr_z9 @kalomaze @antirez @_xjdr @norpadon Trained != served. V3's case for training with MTP is pretty watertight, but deployment is more mixed
you said that you hate speculative decoding so i was wandering why since my understanding is that it was already heavily used in production by deepseek, GLM and al with MTP (and other US lab like google ect..). we both agree that deepseek new method seems better (see my post above!) and this was not my question but nw
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon Whale's method handles verification in a much better manner If mtp 3/5 is used then thruput degrades due to verification overhead
@eliebakouch @kalomaze @teortaxesTex @antirez @_xjdr @norpadon It does impact model quality I would recommend you to try running with & without MTP and judge the quality
@zephyr_z9 @kalomaze @teortaxesTex @antirez @_xjdr @norpadon i don't get your point, this doesn't impact model quality if you do it properly, and doing inference optimization is independent of model quality