Synthetic pre-pretraining reportedly improves language-model efficiency at scale, without a clear grammar link

In a September 30 preprint, researchers report that briefly training language models on synthetic, non-natural-language data before pre-training improves downstream performance and token efficiency at larger scales. They estimate savings of at least 21B pre-training tokens at the 3B-parameter scale. They find no consistent evidence that the gains come from a grammatical prior, instead linking downstream gains to tasks that improve long-range retrieval.