Muon appears better than AdamW after pretraining but worse after fine-tuning as pretraining data scales up

A researcher whose lab produced the work says Muon appears better than AdamW after pretraining but performs worse after fine-tuning as pretraining data scales up. Catherine Li was set to present the findings at MOSS on October 10. A separate MOSS invitation pointed to discussion about what people can meaningfully contribute without huge resources, including in academia.