Here's a thread of six academic (non-world-record) Modded-NanoGPT optimization results from the past few weeks.
Result #23: Kai Lion and Florian Hübler have contributed a 3075-step run using a row-norm control technique called Muown. This result is notable for its simplicity.
Result #24: @zhiweixux showed that the 2026/05/01 record could be improved by 50 steps by separately tuning the WSD cooldown fractions for the hidden matrix vs. other parameters. Note: This can't be applied to the current record because it uses a power lr schedule.
Here's a thread of six academic (non-world-record) Modded-NanoGPT optimization results from the past few weeks.
Result #23: Kai Lion and Florian Hübler have contributed a 3075-step run using a row-norm control technique called Muown. This result is notable for its simplicity.
Here's a thread of six academic (non-world-record) Modded-NanoGPT optimization results from the past few weeks.
Result #23: Kai Lion and Florian Hübler have contributed a 3075-step run using a row-norm control technique called Muown. This result is notable for its simplicity.