Users praise the paper on model capacity enabling retention of rare task updates amid gradient interference because it lays solid foundations despite many open questions remaining.
Plenty of open questions left about how to choose mixtures for a given scale, transfer to real pretraining, post-training as a separate axes, etc, but this paper lays solid foundations to think about all of these. Link: https://arxiv.org/abs/2605.29548
It also suggests that the classic (learning theory) way to think about model capacity in isolation misses an important part of the story. We ought to think about capacity 𝘳𝘦𝘭𝘢𝘵𝘪𝘷𝘦 𝘵𝘰 task diversity in the training data.
This (+ lots of recent work on fine-grained scaling laws and data mixtures, by us and others) confirms that scale and data composition aren't independent levers, what you can learn is a **joint** function of both.