Rohan Anil flags confirmation that an optimizer family now called Muon delivers efficiency lifts over AdamW on the same order as the earlier Shampoo gains, tracing the idea back to a 2024 conversation with Ilya Sutskever. Public benchmark runs on modded-NanoGPT place Muon ahead of both Shampoo and Adam, with the edge tied to its momentum-plus-orthogonalization approach rather than simpler variants. No information yet exists on behavior at frontier scale or any planned paper release.
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.
Keller Jordan’s public NanoGPT runs show Muon pulling ahead while Shampoo sits roughly midway between Muon and Adam, and Spectral Descent lags by about 2x; the distinction comes from accumulation steps plus Newton-Schulz steps rather than pure per-step updates. How those orderings translate outside speedrun-scale experiments is still open.
Muon has appeared in recent NanoGPT and CIFAR-10 training records, suggesting some practitioners are already swapping it in where the benchmark numbers matter. Whether the same swap pays off in production pipelines or larger pretraining runs is not addressed in the current posts.
Many users welcomed the new optimizer's reported gains over AdamW and the Shampoo family as a welcome shift toward practical optimization advances, while some criticized the announcement as vague posting.
All or nothing aspect means there is no incremental reward which just needs faith.
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.
Would LLMs have figured this out on their own? I am not so sure. The right prior is important.
All or nothing aspect means there is no incremental reward which just needs faith.
All you have to do is believe and listen to the gradients.
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.
@swyx It will get rejected due to lack of novelty. :)
@_arohan_ oh man please publish a. paper sometime!!
@_arohan_ oh man please publish a. paper sometime!!
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.
@jaiselsingh Could be great near a good initialization
@_arohan_ My response to the claim that Muon is a renamed version of Shampoo
I've added two optimizers to the public benchmark:
(1) Shampoo (with its original 1/4 power). (2) Spectral descent, which is equivalent to both Muon(mu=0) and Shampoo(b1=b2=0).
Result: Shampoo falls halfway between Muon & Adam; Spectral descent is ~2x slower.
Thread below 1/6
postconditioning achieved internally
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.
@_arohan_ what’s your take on zeroth-order optimization techniques? :>
@_arohan_ Do you think for different types of data (e.g. coding, literature, what not) there will be different Pareto improving optimizers?
@0xaltyni Data distribution definitely would have some impact on some of the choice/hypers of the optimizer.
@_arohan_ I’m curious abt a CFD like structure where you solve a coarse problem to find the flow structure, then refine where the gradients actually matter so zeroth order global search and then refinement
@a_karvonen I really did like parallax runs on it that was posted recently there.
@_arohan_ Cool! so open source it.
@A_K_Nain @_arohan_ i will accept at @aidotengineer we are slutty like that
@vibhuuuus pls get for poaster sessions
@_arohan_ @swyx Hahah ICLR/NeurIPS instant reject 😂😂😂🤣🤣
@_arohan_ wen core automation nanogpt speed run record?
@_arohan_ someone named a shampoo variant after Ilya saying it with ur own chest is crazy, respect
@_arohan_ adamw had the wrong theory being element wise, muon is better especially with weight decay and other tricks, but I don't think an optimizer improvement alone fixes fundamental limitations in ml theory
@_arohan_ Congrats Looking forward to Conditioning properly
Rohan Anil flags confirmation that an optimizer family now called Muon delivers efficiency lifts over AdamW on the same order as the earlier Shampoo gains, tracing the idea back to a 2024 conversation with Ilya Sutskever. Public benchmark runs on modded-NanoGPT place Muon ahead of both Shampoo and Adam, with the edge tied to its momentum-plus-orthogonalization approach rather than simpler variants. No information yet exists on behavior at frontier scale or any planned paper release.
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.