Integration requires only a single line of code.
Why aren’t Diffusion Language Model smart yet? Lacking stable post training is a major bottleneck!
Meet DiPOD: the tripod for diffusion model post-training.
DiPOD boosts accuracy across reasoning tasks, with Sudoku jumping from 22% to 97%, through a one-line code change.
🧵1/5
Users are excited about DiPOD's simple one-line change boosting diffusion language model reasoning accuracy, calling the research cool and exciting.
No Digg Deeper questions have been answered for this story yet.
Paper: https://arxiv.org/abs/2606.13795 Code: https://github.com/Astro-Eric/DiPOD-release Blog: https://astro-eric.github.io/blogs/dipod/ This is an amazing collaboration with @HavenFeng, @pabbeel, @JiantaoJ, @akanazawa, @nhaghtal
🧵5/5
DiPOD tackles double drift by interleaving the gradient steps with self-distillations. In implementation, this results in adding a regularization term to the original objective, and consistently improves on GSM8K, MATH500, Countdown, and Sudoku.
🧵3/5
DLM post-training is hard because log-likelihood is intractable, and people replace it with proxies. We identify the double drift issue: proxy and drift from log-likelihood, and gradient subsequently drifts from policy gradient.
🧵2/5
DiPOD takes a variational inference perspective, and provides a theoretical framework to analyze policy gradients algorithms for generative models. It could produce useful algorithms in other domains like robotics as well.
🧵4/5
@berkeley_ai
@erichzjiang this is some cool research! diffusion models for language, more than just images
@erichzjiang Congratulations on this exciting work on RL for diffusion LLMs! Our recent paper (https://arxiv.org/pdf/2603.12554) also explores RL post-training for DLMs. Since we study similar benchmarks, it would be interesting to compare approaches and results. Looking forward to further progress!
