Reactions from ranked influencers
7 posts
Also the owl paper from Anthropic.
it’s easy to justify why distilling without logits should only be a small constant factor worse by the way: assume you sample from the teacher at temperature 1. training on that token is just training on the full teacher distribution in expectation, you just see one draw instead of probabilities and since even weak base LMs get test losses around 1.5 nats/token, if you assume they’re roughly calibrated then it’s a decent proxy for next-token entropy also being around 1.5 nats. 1.5 nats entropy is the same uncertainty as choosing uniformly around e^1.5 =~ 4.5 tokens and implies that a sampled token carries at least ~22% teacher probability on average. and since real token distributions aren’t uniform, and actually super head heavy, assuming a toy approximation like geometric/power law shapes puts you around 30%/40% probability average language model outputs are way way more collapsed than this, saw a paper that chatgpt response entropy is something like 0.3, so this should be way higher, but we can stick with 30% as a lower bound so this gets you to a hand-wavy 2-3x efficiency drop from gradient noise here, nothing close to vocab scale or orders of magnitude worse! it would be very surprising if logit distillation was massively worse, and the literature supports this: Sequence-Level Knowledge Distillation — Kim & Rush, 2016 Knowledge Distillation of Black-Box Large Language Models / Proxy-KD — Chen et al., 2024 DeepSeek R1 paper literally claims they do this, and sees a large win without logits Sparse Logit Sampling — Anshumann et al., ACL 2025 etc
yeah i think my information argument is wrong, i should have made one on gradient noise training on single sampled tokens is just a higher variance estimate of the same soft teacher distribution gradient if the teacher's distribution is very concentrated then the extra noise is not huge. and further noise in early/mid in training is massively dominated by other noise, not logit noise anyways 1 − exp(−1.5) ≈ 0.78 added variance per token, so should be less efficient than logit distillation, especially late in training, but good reason to not expect a vocabulary-scale or orders-of-magnitude penalty
@willdepue i definitely wouldnt say marginally less efficient (especially and almost dramatically so for stuff that looks like QAT self-distillation/healing) but yes in expectation
@teortaxesTex i don’t think this is a correct read. its much more about the performance of kimi base in pretraining vs. rl performance or on policy distillation. not to say that it doesn’t matter for RL, im sure it does, but that’s not the significant bit for a model close to 5.6/fable
yeah i think my information argument is wrong, i should have made one on gradient noise training on single sampled tokens is just a higher variance estimate of the same soft teacher distribution gradient if the teacher's distribution is very concentrated then the extra noise is not huge. and further noise in early/mid in training is massively dominated by other noise, not logit noise anyways 1 − exp(−1.5) ≈ 0.78 added variance per token, so should be less efficient than logit distillation, especially late in training, but good reason to not expect a vocabulary-scale or orders-of-magnitude penalty
@willdepue I'm not convinced, especially the part from the average 22-40% probability mass on the sampled token to the clean 2-3x efficiency drop is imho too handwavy