Researcher Reports Negative Results Testing Information Metrics On Language Models
——0——
@andrewgwils hi andrew, I'm running some experiments to understand how useful MDL/epiplexity are in practical LM training scenarios. both @yidingjiang and I think these experiments pretty funny and valuable. lmk if you are interested in giving feedback or proposing new experiments!
Andrew Gordon Wilson@andrewgwils@jiaxinwen22 We discuss the tension between information theory and modern AI phenomena here: https://arxiv.org/abs/2601.03220. The good news is that we can shed light on these phenomena by understanding the role of computation and structural information.
12:38 AM · May 23, 2026 · 365 Views
12:46 AM · May 23, 2026 · 255 Views