The project validates a 2024 prediction by Ilya Sutskever.
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.
Some users welcomed the new optimizer's gains over AdamW as a shift from hype to real breakthroughs and called for open-sourcing, while others dismissed the claims as requiring blind faith without incremental rewards or accused the post.
All or nothing aspect means there is no incremental reward which just needs faith.
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.
All you have to do is believe and listen to the gradients.
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.
@_arohan_ oh man please publish a. paper sometime!!
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.
@_arohan_ Do you think for different types of data (e.g. coding, literature, what not) there will be different Pareto improving optimizers?
@_arohan_ Cool! so open source it.
Would LLMs have figured this out on their own? I am not so sure. The right prior is important.
@_arohan_ moving from architecture hype to optimization breakthroughs is a good trajectory
@_arohan_ Instead of vague-posting... contribute to science!
The project validates a 2024 prediction by Ilya Sutskever.
Ilya (sorry for name dropping) met with me in 2024 and said in a meeting that we can do better than Shampoo family (renamed to Muon and friends).
Now I can say this is very true, there exist an optimizer that shows the same scale of improvements that shampoo shows over adamw on deep learning models.
Fundamental optimization continues to drive progress. Fascinatingly the work to get there is all or nothing. Nothing works under everything works.