Entropy-gated bitstream diffusion matches autoregressive model performance
——0——
Researchers introduce entropy-gated bitstream diffusion, a continuous language modeling technique that operates directly on bitstreams using entropy profiles to focus training. The method outperforms masked and uniform diffusion baselines in evaluations and reaches performance comparable to autoregressive language models under the same settings. A related ICML paper adapts existing autoregressive models to diffusion frameworks through implicit representation alignment.
