/Tech2h ago

ByteDance and Renmin University release iLLaDA, an 8B masked diffusion language model that outperforms Qwen2.5 7B

Story Overview

ByteDance and Renmin University researchers trained iLLaDA from scratch as a fully bidirectional 8B masked diffusion model, keeping the same objective through both pre-training on 12 trillion tokens and later fine-tuning, then reported it beating Qwen2.5 7B on several base-model benchmarks while using grouped-query attention and tied embeddings.

3173102.3K

#501

Original post

Tanishq Mathew Abraham, Ph.D.@iScienceLuvr#613inTech

Improved Large Language Diffusion Models

"We introduce iLLaDA (improved LLaDA), an 8B fully bidirectional masked diffusion language model trained from scratch. For pre-training, iLLaDA scales the corpus to 12T tokens, uses grouped-query attention to reduce cache-style inference memory and tied input/output embeddings to reduce parameter count, and modifies the learning-rate schedule for large-scale training. For post-training, iLLaDA modifies the SFT strategy for variable-length generation and trains on a 25B-token instruction corpus for 12 epochs. For inference and evaluation, iLLaDA uses variable-length generation for efficiency and confidence-based scoring for multiple-choice benchmarks"

"Against Qwen2.5 7B, iLLaDA-Base is slightly stronger on average, while iLLaDA-Instruct still lags behind Qwen2.5 7B Instruct."

1:21 AM · Jun 25, 2026 · 2K Views

Benchmark Edge

Where diffusion training actually helps

The model posts clear lifts over earlier diffusion LLMs like LLaDA and Dream on tasks such as BBH and MATH, yet the instruct version still trails the much larger-data Qwen2.5 baseline on average.

Open Question

Access remains an open variable

The preprint links to a GitHub repo for weights and code but supplies no release date, license, or hosting details, so it is not yet known when or how developers can actually run the model.

Sentiment

Users are excited about ByteDance's iLLaDA 8B diffusion language model outperforming Qwen2.5 because it shows text diffusion advancing and open-source models catching up despite transformer dominance.

Pos

100.0%

Neg

0.0%

3 comments with sentiment.

Cluster Engagement

Digg Deeper

No Digg Deeper questions have been answered for this story yet.