Stanford researchers release paper 'A Bitter Lesson for Data Filtering' showing large models improve without data filtering in high-compute data-scarce regimes
Work targets abundant compute and limited data pretraining scenarios.
#158Saining Xie@SAININGXIE@tatsu_hashimoto @giffmana TheRightWay™ strikes again?
https://arxiv.org/abs/2605.19407 We down-sample the common crawl pool and apply filters on top, simulating a smaller data universe. With enough compute, training on the pool catches up to every filter in DCLM, even when we eval on PPL for a higher-quality, filtered corpus.
Tatsunori Hashimoto@tatsu_hashimotoSome new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
This is the case even for more targeted “bad” data, like shuffling the tokens or even injecting completely random token sequences. We find that adding in very large amounts of fresh “bad” data doesn't hurt (and even helps, for shuffled data), compared to more epoching.
Tatsunori Hashimoto@tatsu_hashimotohttps://arxiv.org/abs/2605.19407 We down-sample the common crawl pool and apply filters on top, simulating a smaller data universe. With enough compute, training on the pool catches up to every filter in DCLM, even when we eval on PPL for a higher-quality, filtered corpus.
@sainingxie @giffmana This set of results would argue even more strongly for things like NoFilter, in that in sufficiently high compute regimes for LMs, you may actually get benefits on the majority group (the 'high quality filtered set' we eval on) by including the minority group.
Saining Xie@sainingxie@tatsu_hashimoto @giffmana TheRightWay™ strikes again?
#437Xiao Ma@INFOXIAO@tatsu_hashimoto @ChengleiSi you lost me at "with enough compute"
Tatsunori Hashimoto@tatsu_hashimotoSome new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
#442CLS@CHENGLEISIThings are weird in the (severely) data-constrained regime. Tatsu is always thinking far ahead about the future!
Tatsunori Hashimoto@tatsu_hashimotoSome new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
#1460Jiaxin Wen@JIAXINWEN22Multi-epoch pre-training should be the default setting for pre-training papers
Tatsunori Hashimoto@tatsu_hashimotoSome new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
@tatsu_hashimoto Very cool. I am now expecting a flood of models on AI x bio that try to do the same thing (TBF they are already largely doing this with little success), without realizing at what scale & problem definitions this actually works. At least, I know who to blame.😆
Tatsunori Hashimoto@tatsu_hashimotoSome new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
@tatsu_hashimoto More seriously, I wud love to understand whether this claim holds for bioAI models and applications like DNALMs & single cell FMs that have enough training data but really struggle to learn effectively.
Anshul Kundaje@anshulkundaje@tatsu_hashimoto Very cool. I am now expecting a flood of models on AI x bio that try to do the same thing (TBF they are already largely doing this with little success), without realizing at what scale & problem definitions this actually works. At least, I know who to blame.😆