New Research Uncovers Mode-Hopping Dynamics In Language Model Pre-Training

On benchmarks with an obvious (but wrong) shortcut, this very cool blog shows models hop between generalizing and using shortcuts. I was curious what you see across compute optimal models. Interesting case of U-shaped scaling when ICL examples have a successive answer pattern!