Reactions from ranked influencers
2 posts
Would be great if there were consistently community notes saying that training on only outputs (without logits) works well. Or saying that the term "distillation" is now commonly used to include training on samples without logits. Maybe there should be a canonical write up...
How do you distill a model without logits?! Like, why do you guys keep saying this? It's one thing for some non-technical person to keep bashing Chinese labs. But even the technical folks? Why?! If your idea of Chinese models' improvement is because of "distillation (without logits!) using API data" -- (also, note that the reasoning-traces are held out at the server -- you only get a hash from Anthropic) -- and you're a technical person, you're either, 1. Not that technical, or 2. You're trying to get likes on Twitter.
@RyanGreenblatt @NathanpmYoung jfyi a use-case for a consistent note