I believe diffusion LLMs (actual diffusion LLMs, not "block diffusion" SpecDec tacked onto an AR pretrain) won't make any impact until some algorithmic breakthrough around training for retroactive revisions (which diffusion is actually not great at). AR acceleration is too easy
Positive users find AI architecture debates around diffusion LLMs increasingly interesting every week, while negative users dismiss the emphasis on fundamental architecture breakthroughs as bad research taste and mere fetishism.
No Digg Deeper questions have been answered for this story yet.
This is, on priors, a "bad research taste" take btw we routinely get more alpha with objective design rather than fundamental architecture fetishism but diffusion math (like EBMs) is catnip for big brains, they have invested a great deal into this, so I've grown weary of hype
I believe diffusion LLMs (actual diffusion LLMs, not "block diffusion" SpecDec tacked onto an AR pretrain) won't make any impact until some algorithmic breakthrough around training for retroactive revisions (which diffusion is actually not great at). AR acceleration is too easy
@teortaxesTex No cap AI architecture debates getting more interesting every single week
@teortaxesTex