5h ago

Emmy Liu shows large language models acquire skills in consistent sequence during pretraining, progressing from copying and morphology to arithmetic and complex reasoning across Pythia, OLMo, and Amber models

Graham Neubig shared the preprint on cross-family results.

——0——
Original post
Lisan al Gaib#980@SCALING01OPEmmy LiuEmmy Liu@_EMLIU

Copying → morphology/translation → basic arithmetic → complex reasoning & math. Across every model family we tested, LLMs acquire skills in roughly the same order during pretraining. Can we use this to predict what a model will learn next, just from its internals? 🧵

8:14 AM · May 20, 2026 View on X
QUOTE POSTGraham Neubig#99Graham Neubig@GNEUBIG

Check out our new work on examining what LLMs learn and when!

We posit that LLMs have an implicit curriculum where they learn gradually more complex skills, and attempt to uncover some details of how this curriculum develops over time across model families.

Emmy LiuEmmy Liu@_emliu

Copying → morphology/translation → basic arithmetic → complex reasoning & math. Across every model family we tested, LLMs acquire skills in roughly the same order during pretraining. Can we use this to predict what a model will learn next, just from its internals? 🧵

3:14 PM · May 20, 2026 · 15.3K Views
4:19 PM · May 20, 2026 · 2.7K Views
Emmy Liu shows large language models acquire skills in consistent sequence during pretraining, progressing from copying and morphology to arithmetic and complex reasoning across Pythia, OLMo, and Amber models · Digg