Berton Shares Beyer Lecture Plot on SSL Limits
Ari Morcos replies that the SSL plot shows bad filtering, not a reason to skip it.
Gabriele Berton posted about Lucas Beyer's lecture on vision models. He highlighted a plot where SSL plus a few labeled samples beat pure SSL, shifting his focus toward cheap labels. Ari Morcos replied that the result shows poor filtering rather than a reason to stop filtering. Morcos added that overfiltering removes useful data from many public datasets and stressed the need for curation that preserves non-English capabilities.
Finally watched Lucas Beyer's lecture on "Vision in the Age of LLMs" What I found most interesting: 1. This plot is why he gave up on SSL. They trained SSL vs SSL + few labeled samples, the latter won by far. So his focus became how to get cheap labels and use them well [1/3]
Reactions from ranked influencers
10 postsThis is more an example of a bad filter rather than why you shouldn't filter. That said, I absolutely agree with @giffmana that overfiltering is commonn and that lots of highly useful data is filtered out in most publicly curated datasets. There other public examples of this:…
2. He argues against filtering data used to train VLMs CLIP (and many following papers) filter out non-English captions, so the model is biased towards English-speaking-countries knowledge For example, the data will contain many more landmarks in Canada than in Brazil. [2/3]
This example also highlights the perils of poor filtering, which can easily introduce unintended bias into models, and make them perform disparately for different groups of people. This is why curation to focus on non-English capabilities and populations is so critical if we…
2. He argues against filtering data used to train VLMs CLIP (and many following papers) filter out non-English captions, so the model is biased towards English-speaking-countries knowledge For example, the data will contain many more landmarks in Canada than in Brazil. [2/3]
@arimorcos @giffmana Agree a ton on the selection and sources part :) Our DCVLM’s findings resonate most of what you suggested:
Finding 1: Data filtering barely helps for VLMs! 🤯 Tested 60+ filter configs (CLIP-score, DCLM-fasttext, Nemotron classifiers, UniFilter, perplexity). Best case: +0.8pp over no-filtering. Most filters do nothing or actively hurt! @giffmana @tatsu_hashimoto you might like this!
@giffmana Just dataframe.sample() over and over again until it starts looking clean
+1 to framing problem the data selection as opposed to data filtering - this is a very important distinction I think the whole reason generic data filters have often failed with more scale is that in the good-at-everything model setting, when you have the compute to train on…
Filters for 100% obvious garbage like "unknown.jpg" make sense, but many things that may look like obvious garbage actually aren't, for example "2012-03-21.jpg". For any "smarter" filtering, i will disagree except in two scenarios: (1) you simply can't train on everything or…
also, in general, the other reason I think filters get a bad rap is that the field often bakes in our human-biased intuitions for data quality into filters however, time and time again, machine learning and, in particular, deep learning has shown that what data looks good to a…
