Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.
i think about this almost every day, this is very much against my prior about distillation being important in pre training (for compute efficiency mainly but also for certain capabilities), chinese models are the perfect example of the opposite https://twitter.com/oneill_c/status/2075634892252299415
this is a really interesting analysis - maybe distillation actually doesn’t matter much for the chinese labs https://twitter.com/oneill_c/status/2075634892252299415
The analysis suggests distillation functions mainly as a reinforcement learning warm start.
Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.