/TECHStory update pending

Prime Intellect research lead says QwQ is a better teacher model than DeepSeek-R1 for distilling Qwen2.5-7B

Alex Dimakis notes that shared pretraining data distributions improve distillation.

Story Brief

Alex Dimakis notes that shared pretraining data distributions improve distillation.

Commentary on X

Highest ranked

@willccbb Thanks! I believe the best teacher is one that is good but also close to the pretraining data distribution of the student. They speak the same language.

will brown@willccbbTECH