Microsoft's Luca Soldaini defended data inspection as a vital skill.
Jerry Liu#733
Bad take, you absolutely need to look at the data, manually
Just learned:
Software engineers used to do manual data labeling at Scale AI while Alex Wang was CEO. After he left, new leadership joined, and were HORRIFIED to learn this. Stopped it ASAP
Now at Meta, software engineers are assigned manual data labeling... see the pattern?
Positive users describe Meta's reassignment of engineers to manual data labeling as meditative, fun, and highly rewarding, while negative users sarcastically dismiss the emphasis on its importance.
More context from Scale:
1. It’s not exactly about new leadership: but when Meta “bought” or “bought out” Scale and leadership changed, the focus drastically changed from devs doing manual data labelling to doing dev work
2. Talked w Scale leadership: they say in the early days data labelling was very important for the company, it was thus both necessary and highly rewarded - and this was before the Meta “sale”
3. Devs don’t like doing manual data labelling. But it’s something that creates great value when you are either selling data, or data labelling, or need high-quality data. Scale was (is?) in this business, and Meta clearly wants to create a SOTA coding model
Just learned:
Software engineers used to do manual data labeling at Scale AI while Alex Wang was CEO. After he left, new leadership joined, and were HORRIFIED to learn this. Stopped it ASAP
Now at Meta, software engineers are assigned manual data labeling... see the pattern?
There are so many stories like this, about how important unferstanding data quality is for model performance. I do think its a really hard skill to transfer, probably why everyone cares so much about "taste" and "vibes" right now
Bullish for Meta Liang Wenfeng *personally* annotated data for V3 taste is something you've got to demonstrate by example, and it's not just glamorous architecture designs.
not sure i see the issue here
Just learned:
Software engineers used to do manual data labeling at Scale AI while Alex Wang was CEO. After he left, new leadership joined, and were HORRIFIED to learn this. Stopped it ASAP
Now at Meta, software engineers are assigned manual data labeling... see the pattern?
@FrostForger evals are data RL envs are data model outputs are data
kernels are important tho 😉
@soldni i have au tism
(tbh i find data a meditative and fun and interactive puzzle, as long as you dont stress over how it's "menial and dead end" you can enjoy it)
@pjreddie i wonder if there are a few enterprising individuals that that train custom models to surreptitiously do the labelling until shogun wang tells them to seppuku
@soldni but those are ml people who look at data out of theor own voilition, not forcibly reassigned from another org
@soldni 1. its all testing. 2. its all kernels
@jerryjliu0 wait, what issue were they trying to point out?
@Skiminok @ArneKoehn data labeling, or creating labeled data?
@yoavgo @ArneKoehn I understand where you're coming from (and agree with that charitable view) – but it's literally the situation. About 5000 regular Meta SWEs with past "exceeds expectations" ratings have been forcefully reassigned to a new org where data labeling is their *only* job.
@yoavgo @ArneKoehn The latter. Creating SWE-related RL environments and tasks.
@Skiminok @ArneKoehn seems very legit to me, its a software engineering project.. no?
@yoavgo @ArneKoehn Data quality from a deliberately hired and motivated org of experts (Anthropic model; or countless modern data vendors) >> forced reassignment where the company is just waiting for people to resign.
@Skiminok @ArneKoehn i dont understand why this specific reassignment is so much worse than other ones. data pipelines are important and interesting. RL pipelines likewise.
@yoavgo @ArneKoehn Data quality from a deliberately hired and motivated org of experts (Anthropic model; or countless modern data vendors) >> forced reassignment where the company is just waiting for people to resign.
@soldni cannot be overstated
I am begging you to look at your data. Please look at the data
evals worse than expected? look at the data evals better than expected? *definitely* look at the data evals about what you expected? believe it or not ....
@menhguin same
@GergelyOrosz The pattern is that AI work keeps moving the bottleneck back to human judgment. Labeling, review, evals, cleanup - none disappear just because the org chart changed.
@soldni looking at data imo is so easy and has so much reward imo
@jerryjliu0 depends what you're testing under. lab conditions are clean. real world isn't.
accent the model wasn't trained on. construction site with rust and mud. flash crash nobody modeled.
that's where the issue lives.
@jerryjliu0 Manual labeling only got embarrassing after leadership changed. Humans hate grunt work most when someone else names it.
Microsoft's Luca Soldaini defended data inspection as a vital skill.
Jerry Liu#733Bad take, you absolutely need to look at the data, manually
Just learned:
Software engineers used to do manual data labeling at Scale AI while Alex Wang was CEO. After he left, new leadership joined, and were HORRIFIED to learn this. Stopped it ASAP
Now at Meta, software engineers are assigned manual data labeling... see the pattern?