ZyphraAI converted its ZAYA1-8B-base autoregressive model to a diffusion LLM through mid-training rather than training from scratch. The company applied a TiDAR-based diffusion-conversion step followed by diffusion supervised fine-tuning on its existing stack. The resulting model diffuses 16-token blocks in a single step from a mask prior, matches autoregressive logits via speculative decoding, and mixes logits during sampling to deliver speedups over earlier methods such as TiDAR. Platform discussion noted diffusion language models increasingly adopting autoregressive traits on smaller sequential token blocks.
Ben (no treats)#1623
We present ZAYA1-8B-Diffusion-Preview, the first diffusion language model trained on @AMD.
Autoregressive LLMs generate one token at a time; diffusion generates a block in parallel, speeding up inference.
We show a 4.6-7.7x decoding speedup with minimal quality degradation 🧵
Training a diffusion LLM from scratch is challenging, and diffusion inference advantages don't show up at training.
So we convert instead. Building on the TiDAR recipe, we take ZAYA1-8B-base and run diffusion-conversion mid-training plus diffusion-SFT on our existing stack.
We diffuse 16-token blocks in one step from a mask prior. Tokens matching AR logits are accepted via standard speculative decoding. Our sampler mixes AR and diffusion logits for acceptance. Unlike TiDAR, this yields further speedups at some quality cost.
Yuxi keeps winning
We diffuse 16-token blocks in one step from a mask prior. Tokens matching AR logits are accepted via standard speculative decoding. Our sampler mixes AR and diffusion logits for acceptance. Unlike TiDAR, this yields further speedups at some quality cost.
The DeepSeek of the West, Zyphra, has been releasing a series of AI models that are much more compute-efficient than the SOTA, and they are open weight / open source.
We present ZAYA1-8B-Diffusion-Preview, the first diffusion language model trained on @AMD.
Autoregressive LLMs generate one token at a time; diffusion generates a block in parallel, speeding up inference.
We show a 4.6-7.7x decoding speedup with minimal quality degradation 🧵
That's a cool way to explain the logistical salience of the trick, but let's be honest, such "diffusion endpoint" is just the logical endpoint of speculative decoding for AR. Pure diffusion language models had another… more confused promise.
Diffusion is the endpoint of inference. Decode becomes as efficient as training. Everything sits on the roofline. Pure FLOPs will be the only bottleneck.
Excited to push the frontier on language diffusion. Massive congrats to the team and an exciting time ahead.
@teortaxesTex >ask the performant dlm author if its seqlevel denoising or block-causal >"its a good diffusion language model, sir" >look inside >its block-causal
Yuxi keeps winning
Cool.
We present ZAYA1-8B-Diffusion-Preview, the first diffusion language model trained on @AMD.
Autoregressive LLMs generate one token at a time; diffusion generates a block in parallel, speeding up inference.
We show a 4.6-7.7x decoding speedup with minimal quality degradation 🧵
Beyond raw speed, diffusion enables more expressive generation than left-to-right autoregression, and cheaper on-policy rollouts, meaning more RL and test-time-compute scaling per dollar.
A fully post-trained diffusion model is coming soon.
Blog: http://zyphra.com/post/zaya1-8b-diffusion-preview
We trained and performed the diffusion conversion on AMD. The greater VRAM capacity of AMD and our FLOP and memory-efficient attention variant CCA was instrumental for efficient training given the longer sequence length and other overheads of diffusion.
CCA plays a vital role here by being the only dense attention block that reduces the FLOPs in attention. Diffusion makes decoding compute-bound like prefill so CCA lets us decode more tokens in parallel before becoming compute-bound, directly resulting in decoding speedup.
@ZyphraAI is an open superintelligence research and product company based in San Francisco, CA on a mission to build human-aligned AI that helps individuals and organizations reach their fullest potential.
Apply to join us! https://jobs.ashbyhq.com/zyphra
@teortaxesTex Is there any way to run these models using llama.cpp?
