/Tech13h ago

DiffusionGemma Bypasses Sampling Wall With Loopholed Discrete Diffusion

6120157313.8K
Caglar Gulcehre#878
Original post
Oussama Zekri@oussamazekri_

Okay, even more interesting : DiffusionGemma is a “loopholed” diffusion model!

Discrete diffusion usually hits the sampling wall:

the model has a rich distribution over tokens,
then at each step, sampling crushes it into one hard token.

A lot of previously computed belief disappears. But DiffusionGemma keeps the previous logits alive.

So it denoises from the token
AND from the belief behind the token.

That’s the idea behind « Loopholed Discrete Diffusion», a paper I was playing with this week. Exciting to see this at scale !

Oussama Zekri@oussamazekri_

So DiffusionGemma is a 26B uniform-state diffusion language model! Very interesting.

This might be one of the first open-weight releases of this kind at this scale. (Not forgetting @dvruette’s public 10B version)

11:55 AM · Jun 10, 2026 · 12.5K Views
Sentiment

Users praise the DiffusionGemma paper for its loopholing approach to discrete diffusion sampling and thank the coauthors for their contributions and ideas.

Pos
100.0%
Neg
0.0%
2 comments with sentiment.
Cluster Engagement
Posts from X
Most Activity
Most Activity
VIEWS468
Oussama Zekri@oussamazekri_

Indeed!

https://github.com/huggingface/transformers/blob/main/src/transformers/models/diffusion_gemma/modeling_diffusion_gemma.py#L761-L794

14hViews 468Likes 2
BOOKMARKS5LIKES8
Oussama Zekri@oussamazekri_

Super nice paper by @pyross0000 @jaesikyoon_ @jdeschena @caglarml @SungjinAhn_

https://arxiv.org/abs/2510.19304

14hViews 446Likes 8Bookmarks 5
RETWEETS2
Mingyu Jo@pyross0000

DiffusionGemma uses the core mechanism of Loopholing, our ICLR 2026 paper!

Discrete diffusion hits a sampling wall: rich token beliefs collapse into one hot token at every step.

Loopholing bypasses this with a deterministic latent pathway, improving generation quality.

5hViews 1.2KLikes 39Bookmarks 11
REPLIES1
Fatih⏩⤴️@taskinfatih

@oussamazekri_ @jdeschena Is it bidirectional?

12hViews 190Likes 1
Oussama Zekri@oussamazekri_

@taskinfatih @jdeschena Yes!

12hViews 150Likes 1
Mingyu Jo@pyross0000

Grateful to all coauthors @jaesikyoon_ , @jdeschena , @caglarml , and @SungjinAhn_ for the ideas, discussions, and support that shaped this work.

5hViews 40Likes 1
Mingyu Jo@pyross0000

Paper: https://arxiv.org/abs/2510.19304 Project: https://sites.google.com/view/lddms/home Code: https://github.com/ahn-ml/lddm

5hViews 31