On Friday, we released six new state-of-the-art drafters for accelerated inference.
We also put out a blog post on why spec dec is so great. Supporting that was a roofline model of speedup from speculation.
Play with it in our LLM Engineer's Almanac:
https://modal.com/llm-almanac/spec-dec-roofline
No Digg Deeper questions have been answered for this story yet.
Read about how this performance model works (what is assumes, where it applies, and where it fails) and why spec is the "Bitter Lesson-pilled" approach to inference performance here:
https://modal.com/blog/spec-is-all-u-need
On Friday, we released six new state-of-the-art drafters for accelerated inference.
We also put out a blog post on why spec dec is so great. Supporting that was a roofline model of speedup from speculation.
Play with it in our LLM Engineer's Almanac:
https://modal.com/llm-almanac/spec-dec-roofline