Users appreciate the Microsoft report showing less LR decay boosts LLM post-RL performance because it highlights the value of studying hyperparameters and LR schedules.
Now, why do lower-curvature models exhibit all those favorable properties? IMO, we still lack a completely satisfactory answer to that question. To say "flatter minima = better" is too simplistic; for example, when fine-tuning, people usually report that higher LR's are *bad*
For example, in this figure from https://arxiv.org/abs/2103.00065, you can see that cutting the learning rate during gradient descent training causes the curvature (top Hessian eigenvalue) to immediately rise:
Background: in old-school vision tasks like ImageNet/CIFAR, it was widely observed that larger learning rates often yielded better generalization performance than smaller ones. But, for a long time, it was unclear whether an analogous point held for LLMs. https://arxiv.org/abs/1711.04623
That was for an MLP, but the same thing happens in LLM training: learning rate decay causes curvature to grow.
(figures from: https://arxiv.org/abs/2604.13627, https://arxiv.org/abs/2510.06213, https://arxiv.org/abs/2603.16127)
For a way to quantify this effect precisely, see: https://arxiv.org/abs/2410.24206
It seems from a number of recent works that the answer is yes: * Watts, Li et al (https://arxiv.org/abs/2605.02105) found that pretraining with a larger peak LR, or a shorter LR decay period, causes the model to better retain its pretraining performance after fine-tuning.
Why does the learning rate used during training affect the resulting model? The answer seems to revolve around loss function curvature: large-LR optimizer dynamics implicitly regularize the curvature of the loss, preventing the optimizer from moving into high-curvature regions.
* Yano et al (https://arxiv.org/abs/2603.16127) found that learning rate decay during pretraining hurts performance on evals after SFT.
* Catalan-Tatjer et al (https://arxiv.org/abs/2510.06213) found that pretraining with a larger peak LR resulted in better performance after quantization, and that post-quantization performance degrades during LR decay.
Nevertheless, hyperparameters matter, and I'm glad we're starting to see good science about LR schedules for LLMs.
PS: as noted by Catalan-Tatjer et al, weight-averaging recovers many benefits of LR decay but without increasing sharpness. More should consider weight-averaging!