Work targets abundant compute and limited data pretraining scenarios.
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
Positive users find the Stanford study on large models benefiting from unfiltered data fascinating since it suggests compute outperforms filtering efforts, whereas negative users dismiss it as a scaling trap that conceals bad data.
Explain this in plain English for coders that are using LLMs to code
When training or fine-tuning LLMs on limited high-quality code data but with plenty of compute, skip aggressive filtering of noisy, low-quality, or even shuffled snippets. Large models tolerate—and often improve from—the extra volume because it acts as built-in regularization that curbs overfitting. For coders, this means throwing more raw GitHub dumps or mixed examples into your dataset instead of hand-curating perfect ones.
Interesting results, was debating with ppl on this many times.
Duplication (multi-epoching or duplicated data) hurts training a lot more than some bad data for pretraining
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
I feel this conclusion might be bit misleading. DCLM-baseline is 3.8T. If you add 1T of low quality data and do more epochs, noise will eventually ironed out and more compute will seem to generate same result as original 3.4T giving you false assurance that filtering is not needed.
But if you hypothetically had 4.8T of same good quality data and spent same compute on that, you will get much better results with less compute.
In DCLM paper they had already observed that gains don’t improve after 3.8T. So, you can use more data and spend more compute but your perf remains same even with more compute.
Another interesting fact is that Nemotron-CC created 6T of more unique tokens but they still didn’t beat DCLM. However, a much smaller HQ set with synthetic rewrites did.
Both pointing to the fact high quality data will always beat low quality data, token-per-token and flop-per-flop. If you can’t separate out low quality data AND if good quality data is still in majority, more compute will eventually will take care of low quality tokens.
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
Multi-epoch pre-training should be the default setting for pre-training papers
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
https://arxiv.org/abs/2605.19407 We down-sample the common crawl pool and apply filters on top, simulating a smaller data universe. With enough compute, training on the pool catches up to every filter in DCLM, even when we eval on PPL for a higher-quality, filtered corpus.
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
I love the work a lot, but most of the time people are still under budget, and recently more so in post-training like RL.
When each rollout is noisy and takes a lot of money and time, filtering good ones cleverly can be much better than scaling up (which we cover in RAGEN-2).
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
@tatsu_hashimoto @giffmana TheRightWay™ strikes again?
Ok this makes me super happy.
The "NoFilter" work, paper, and advocacy that @angelinepouget and I argued so hard for is bearing fruit! Looks like it made MetaCLIP team massively change direction for their just released MetaCLIP2.
This is now TheRightWay™, and thanks for the prominent ack in the paper🤗 Very eager to dig into that dataset!
@tatsu_hashimoto Very cool. I am now expecting a flood of models on AI x bio that try to do the same thing (TBF they are already largely doing this with little success), without realizing at what scale & problem definitions this actually works. At least, I know who to blame.😆
This is the case even for more targeted “bad” data, like shuffling the tokens or even injecting completely random token sequences. We find that adding in very large amounts of fresh “bad” data doesn't hurt (and even helps, for shuffled data), compared to more epoching.
https://arxiv.org/abs/2605.19407 We down-sample the common crawl pool and apply filters on top, simulating a smaller data universe. With enough compute, training on the pool catches up to every filter in DCLM, even when we eval on PPL for a higher-quality, filtered corpus.
Things are weird in the (severely) data-constrained regime. Tatsu is always thinking far ahead about the future!
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
@anshulkundaje I think part of this is an argument along the lines of "even low-quality data has *some* structure, and that is better than using more weight decay". In practice, you'd rather spend your time doing data augmentation and so on first, and even then, there is a hard limit..
@tatsu_hashimoto More seriously, I wud love to understand whether this claim holds for bioAI models and applications like DNALMs & single cell FMs that have enough training data but really struggle to learn effectively.
@tatsu_hashimoto There was this recent paper [1] by @mmbronstein issuing a similar clarion call. Fascinatingly they cited an a16z podcast episode with Aviv Regev quite literally titled "quantity is quality" :) [1] https://pubs.rsc.org/en/content/articlelanding/2026/sc/d6sc01189f#cit34
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
@Guodzh There are a lot of methodological problems with the experiments. Enough that I don’t think any meaningful conclusions can be drawn either for or against its hypothesis.
Interesting results, was debating with ppl on this many times.
Duplication (multi-epoching or duplicated data) hurts training a lot more than some bad data for pretraining
@sainingxie @giffmana This set of results would argue even more strongly for things like NoFilter, in that in sufficiently high compute regimes for LMs, you may actually get benefits on the majority group (the 'high quality filtered set' we eval on) by including the minority group.
@tatsu_hashimoto @giffmana TheRightWay™ strikes again?
@tatsu_hashimoto More seriously, I wud love to understand whether this claim holds for bioAI models and applications like DNALMs & single cell FMs that have enough training data but really struggle to learn effectively.
@tatsu_hashimoto @ChengleiSi you lost me at "with enough compute"
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
@tatsu_hashimoto Very interesting paper! Interestingly, in self-play RL with purely synthetic data, we have the opposite conclusion. Paper dropping soon.
Some new results I found surprising that I’m tweeting for Chris (who isnt on here). With enough compute, the best data filter for LMs (on DCLM) might be no filter. Why? Large models can tolerate a surprising amount of nominally 'low quality' data, and can sometimes even benefit.
@PandaAshwinee @ChengleiSi I think the regime where this is true is very far out on the compute scales. It's after you've exhausted ensembling / synth data / etc. Even in the "naive" case where you dont do this, we don't expect to see these effects for several orders of magnitude more compute, not 27B.
@tatsu_hashimoto @ChengleiSi not sure i agree -we’re going to post results soon showing that 7B “doesn’t benefit as much” from filtering (on DCLM) vs 1B, yes, but i wouldn’t extrapolate that trend out to expect “no improvement” at 27B
Does this transfer to benchmarks? The downstream evals are much noisier, but the trends are generally in line with what we see in the pretraining case, with the pool starting low and eventually catching up.
@tatsu_hashimoto Interestingly, this scenario is quite naturally observed in mammalian genomes littered with repeats & huge swaths of neutral arbitrary DNA. Wud love to discuss this more (y this works for NLP but not DNA).
For these scaling trends, what would be the implied crossing point where an internet-sized pool would benefit from not filtering? The projections suggest ~1e30 FLOPs. Much larger than the largest training runs today, but perhaps within reach in a few years.
