Andrew Gordon Wilson says Shampoo's Modded-NanoGPT speedrun gains rely on an undocumented stability flag

Result #35: @_arohan_ has achieved a significant >700-step improvement to the best known DistributedShampoo config.

This was achieved by switching to one-sided Shampoo, numerically stabilizing the run using an undocumented pseudoinverse flag, and retuning other hparams. 4/5

I created this new speedrun track, which compares results in terms of steps rather than wallclock, specifically to give a fair chance to optimizers other than Muon.

Happy to see the resulting accumulation of public knowledge!

Result #33: @varunneal has provided a PSGD-Kron (Pooladzandi & Li 2024; Li 2024; Li 2022; Li 2018; Li 2015) run! It uses hyperball optimization. 3/5

Result #36: @konstmish improved the best known hyperparameters for the Muon + aux-AdamW baseline. The improvement came mainly from AdamW hparams.

Result #37: @wen_kaiyue then grafted that change onto Muon-Hyperball. Both runs improved by 75 steps. 5/5

Result #32: @mihai673 has achieved a 30-step improvement over the old 2026/05/09 record by adding a SODA (Pethick et al. 2026)-style anchor towards init.

It is unknown whether this technique can also improve the current record. 2/5

@DimitrisPapail https://arxiv.org/abs/2507.07101

@andrewgwils Small Bs all the way doesn’t work but adaptive works reasonably well eg low first high later.

@andrewgwils Also

@kellerjordan0 steps-based leaderboard is such a clean framing, curious if itll make Muon folks pivot their strategy