How optimizer switches between pre-training and fine-tuning might affect benchmark scores

A commenter wonders whether the work tested different optimizers during pre-training and supervised fine-tuning. They cite Moonshot's Muon paper for benchmark losses after an optimizer switch, but guess that switching between matrix optimizers might have less effect. They note that the pre-training optimizer is unknown for the model in the ablation they discuss, and don't think this changes the main transfer result. In a later post, they add, “forgot it was lora.”