Report
A preprint tests synthetic pre-pretraining against natural-language training
Researchers report that repeated training on a 100M-token natural-language corpus beat every synthetic curriculum in their compute-matched tests.
TLDR
The team compared pre-pretraining on synthetic distributions, including k-dyck and NCA, with training solely on natural language at matched compute. They say the synthetic approach never outperformed natural-language-only training, even when language was scarce. With a 100M-token corpus, repeated passes over natural language beat every synthetic curriculum they tested.
Combined views
675
3 Sources, first seen ago
