@AnimaAnandkumar no FP32 master weights is the real headline here. that's a massive memory win if you're running training infra

Excited to share our @icmlconf paper: M+Adam: Low-Precision Training via Additive–Multiplicative Optimization We introduce M+Adam, an optimizer that combines Adam-style additive updates with Madam-style multiplicative updates to enable more effective optimization of low-precision master weights. Modern training already uses BF16, FP8, and FP4 arithmetic, but typically retains higher precision master weights so that small optimizer updates are not lost. We instead study optimization directly on BF16, FP8, and NVFP4 master weights, where the optimizer must operate under the constraints of the low-precision floating-point grid. Additive and multiplicative updates have complementary strengths. Additive updates naturally handle small weights, exact zeros, and sign changes. Multiplicative updates remain effective at large magnitudes, where additive updates can be rounded away. Paper: https://arxiv.org/abs/2607.10611 Code: https://github.com/Anima-Lab/M-Adam-Low-precision-training @SLoeschcke @Caltech

Across transformer models up to 1B parameters and training budgets from 1x to 8x Chinchilla, M+Adam consistently improves over AdamW across low-precision master-weight settings. The gains are largest in the most aggressive low-precision regimes. Low-precision training requires optimizers whose update geometry matches the floating-point grid. M+Adam enables end-to-end low-precision training without higher precision master weights.

M+Adam applies multiplicative and additive updates in a single optimizer step, combining scale-aware progress (multiplicative) with local corrections (additive). https://x.com/AnimaAnandkumar/status/2076856740507992422/photo/1