A Microsoft Research AI Frontiers principal researcher has posted charts from small overnight runs showing that heavily tuned AdamW and WaveAdam variants can reach validation loss levels per step comparable to the Muon baseline on FineWeb, while noting that AdamW still performs more serial updates inside each step.
May be possible that uber optimized Adam can be match val/step stock Muon.
Timing wise, we have a little problem because of small batch early on for Adam as the AdamW “step” is token-equivalent to Muon’s step but each step for AdamW have more serial updates
I'm weak. I switched to Adam.😭
The same plots reveal that any per-step parity may come at the cost of longer wall-clock time early in training, leaving practitioners to weigh whether the extra serial work is worth the tuning effort.
All reported comparisons use modest overnight runs on limited hardware, so it remains unclear how the variants behave at larger model scales or across multiple random seeds.
@DimitrisPapail Release the Nesterov Adam sir
May be possible that uber optimized Adam can be match val/step stock Muon.
Timing wise, we have a little problem because of small batch early on for Adam as the AdamW “step” is token-equivalent to Muon’s step but each step for AdamW have more serial updates
@DimitrisPapail The whole 'machine god SGD' to 'I switched to Adam' arc is too relatable. If AdamW matches per step but still loses on timing, isn't that kind of Muon's whole point?