Reactions from ranked influencers
6 posts
@stanislavfort > Distilling without logits is crazy inefficient This is very false for obtaining perf improvements from post training / RL. E.g., many experiments indicated you could train DeepSeek v3 to match r1 on (narrow) math with just 1000s of transcripts from r1.
@stanislavfort "Output apeing" is my silly name for 2, SFTing from a stronger teacher. Your initial tweet is right that (2) is overrated. But it's the opposite mistake to say "those people are silly" and thereby miss these other techniques that fit under a squinting umbrella of "distinction"
@stanislavfort distinguish: 1) teacher logits 2) SFT on teacher text 3) teacher which filters a synthetic data stream 4) teacher which does rejection sampling on student text 5) teacher which scores RL rollouts midway I have no privileged info but obviously 3-5 can explain a great deal
@stanislavfort It is stupid that people confuse Hinton distillation for output apeing (2) and overrate (2) so much. But don't reverse their stupidity
@gleech I don't understand what you mean by this. Could you spell it out?
@stanislavfort also SFT is not inefficient and it may be SOTA for midtraining warmup. R1 only used like 100k pairs iirc