Speculation Is All You Need.
In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x.
Over 1k output tps for 3.5 122B-A10B on a B200.
Read the blog for why we're all-in on spec dec.
https://modal.com/blog/spec-is-all-u-need
No Digg Deeper questions have been answered for this story yet.
figures in this blog post go pretty hard imo
Speculation Is All You Need.
In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x.
Over 1k output tps for 3.5 122B-A10B on a B200.
Read the blog for why we're all-in on spec dec.
https://modal.com/blog/spec-is-all-u-need
The tl;dr: speculation is "Bitter Lesson-pilled". You can ~linearly increase generation speed by increasing acceptance length.
Increasing acceptance length is a machine learning problem. And ML is scaling rapidly.
Speculation Is All You Need.
In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x.
Over 1k output tps for 3.5 122B-A10B on a B200.
Read the blog for why we're all-in on spec dec.
https://modal.com/blog/spec-is-all-u-need
uvx modal setup && uvx modal shell --gpu b200
Yeah let me spin up my b200….
Third, with a roofline model, building off the great work of @finn_fergus.
This interactive model is available in Modal's LLM Engineer's Almanac.
https://modal.com/llm-almanac/spec-dec-roofline
Second, with a toy model of speedup:
We, along with @zhijianliu_ and @jianchen1799, are excited to share these models on @huggingface for the entire community to use.
We look forward to getting speculative decoding speedups into the hands of more people as they start to own their inference.
More on that soon :)
Third, with a roofline model, building off the great work of @finn_fergus.
This interactive model is available in Modal's LLM Engineer's Almanac.
https://modal.com/llm-almanac/spec-dec-roofline
In addition to the empirical data in that chart, we walk through three ways to build your intuition and understanding of just how good speculative decoding is.
First, mocking acceptance with @sgl_project:
The tl;dr: speculation is "Bitter Lesson-pilled". You can ~linearly increase generation speed by increasing acceptance length.
Increasing acceptance length is a machine learning problem. And ML is scaling rapidly.
"we have LPUs at home"
You read that right, 1k tps!
Second, with a toy model of speedup:
In addition to the empirical data in that chart, we walk through three ways to build your intuition and understanding of just how good speculative decoding is.
First, mocking acceptance with @sgl_project:
@charles_irl @Alibaba_Qwen 🐐 will we have a k2.6 drafter soon?
@charles_irl interesting thanks
@charles_irl @Alibaba_Qwen extremely naive and dumb question - would spec dec be useful for audio output? the timing of the correction and rendering the audio seems tricky, but maybe i'm misunderstanding.
@charles_irl @Alibaba_Qwen holy smokes chief
@__smiz @Alibaba_Qwen would definitely be for tokenized audio! im out of date on audio archs unfortunately
@charles_irl Damn and I was just about to add Modal support to gml https://salazar-99.github.io/gml/
@kubedoll @Alibaba_Qwen @_dcw02 doesn't miss
@charles_irl @Alibaba_Qwen @no_stp_on_snek
@hsu_byron @charles_irl @Alibaba_Qwen I would do it for k2.6 and k2.7 in exchange for a kimi labubu
@jamesahou vroom vroom
@hsu_byron @Alibaba_Qwen 🔜
@charles_irl @Alibaba_Qwen Wait what happened to qwen 3.6 27B dflash model? Was it unable to beat the mtp baseline?