@Mascobot why do you guys have such faith in distillation
The time gap and cadence release between SOTA closed models and open source models has looked roughly the same for the past years.
I guess it has to do more with distillation vs anything else.
No Digg Deeper questions have been answered for this story yet.
distillation is a thought-terminating cliche Have you seen research on it? MSFT's Phi series? They were shit, fell apart inches away from benchmarks on which they rivaled 3-10x larger models. Why are Grok, Gemini such slop? Do all PRC labs have a multi-year lead in distillation?
The time gap and cadence release between SOTA closed models and open source models has looked roughly the same for the past years.
I guess it has to do more with distillation vs anything else.
Distillation is important, but it's a small contribution to the overall competitiveness, and might actually be counterproductive at the current frontier. Reverse engineering contributes a lot more.
distillation is a thought-terminating cliche Have you seen research on it? MSFT's Phi series? They were shit, fell apart inches away from benchmarks on which they rivaled 3-10x larger models. Why are Grok, Gemini such slop? Do all PRC labs have a multi-year lead in distillation?