Muon's target loss steps fell to 3,250 after tuning
😂lol now Shampoo is again in the middle between Muon and Adam. next round, who will tune Shampoo to match / surpass Muon?
I just submitted a PR to modded-nanogpt with better hyperparams. With them, Muon can reach the target loss after 3250 steps instead of 3325. Always tune your baseline well when doing research. Weak baselines can make any idea look promising
the grid sweep chud cowers at the sight of the "guess ballpark-ish hparams and pray that they work" lion
tune your hparams hard. RNGesus is waiting for you in the loss valley somewhere.
@kellerjordan0 Codex gave me this estimate: ~77% of the gain came from AdamW hyperparams (make learning rates 1.5 bigger) ~23% came from Muon hyperparams (smaller lr, bigger weight decay)
I didn't touch anything else.
@konstmish Very nice. Would you happen to know how much of the improvement was from tuning the Muon hparams? As opposed to AdamW
@zacharynado Yes, but then it has to be a lot more than a single problem. Different scales, domains, architectures. Like AlgoPerf.
@konstmish really if we are doing this fairly, each algo should have the same tuning budget, defined by number of hparam points tried.
it implicitly punishes algos with more hparams to tune, but imo that's a fair punishment because in reality practitioners only have a limited tuning budget
@YouJiacheng we should give each of them the same tuning budget and find out :) at a certain point we'll just be overfitting to this specific benchmark though
@konstmish really if we are doing this fairly, each algo should have the same tuning budget, defined by number of hparam points tried.
it implicitly punishes algos with more hparams to tune, but imo that's a fair punishment because in reality practitioners only have a limited tuning budget
@YouJiacheng We should try these on a bigger scale we're really overcooking on this problem
Muon's target loss steps fell to 3,250 after tuning
😂lol now Shampoo is again in the middle between Muon and Adam. next round, who will tune Shampoo to match / surpass Muon?
I just submitted a PR to modded-nanogpt with better hyperparams. With them, Muon can reach the target loss after 3250 steps instead of 3325. Always tune your baseline well when doing research. Weak baselines can make any idea look promising