Path dependency can cause varying overconfidence on misclassified data.
I have a silly shower thought:
Say you train machine god and you reach some Val loss you’re happy with. But there are multiple models with the same loss.
Does it have different “personality traits” depending the optimizer? Are aspects of the optimizer path affecting final model behavior?
Is machine god trained with SGD perhaps more chill than with Muon?
Perhaps!
Some users praise a paper on whether optimizer choice shapes AI model behavior, calling it one of their favorites and very cool.
@DimitrisPapail 👀
been a while since i've seen such a well-articulated paper highlighting train-test gap, path-dependency of training dynamics on convergence, and more
it would be a funny stretch if a "better optimizer" now leads to "overconfidence on misclassified test examples", aka brittle sycophancy we now see in many frontier models... 👀
@DimitrisPapail one of my favorite papers of all time https://arxiv.org/abs/2507.12224
@DimitrisPapail yes https://jiaxin-wen.github.io/blog/generalization-dynamics.html
@_ueaj Huh!
Yes, this seems to happen a lot but I am not sure how to characterize. Also if you use one optimizer for pretraining like Adam and then switch it out to something else like matrix aware optimizers, then performance can be poor compared to sticking to Adam. It’s personally a problem I am very interested in understanding how to characterize or at least develop a unified recipe for.
@_TarunKathuria Ha, very cool!