Positive users endorse the argument that distillation offers limited competitive edge for frontier models, while negative users call the claims overblown, overrated, or fearmongering by labs like Anthropic.
Based on 18 visible X reactions from 99 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@oneill_c Is this similar to how Google when they first dropped Antigravity they were serving Sonnet 4 and Opus 4 but it was really 3.5?
@oneill_c It’s so easy that’s why the Chinese prefer to steal.
@oneill_c great take!
@oneill_c this is a really good point
@oneill_c Honestly a very good point
@oneill_c Valuable datapoint 💯
afaik there is only two public mention of knowledge distillation for "frontier model" training: - gemini flash/lite - llama 4 mistral, qwen also used it but for much smaller model both cite the same co distillation paper https://arxiv.org/abs/1804.03235 https://x.com/eliebakouch/status/2075937804148740444/photo/1 https://twitter.com/oneill_c/status/2075634892252299415
i still think anthropic/oai use it, but i can't find any mention for mistral, the ceo also mention in a itw that they are/want to train bigger model internally and distill it to serve to customer
also i just remembered that llama Behemoth was a 2T, 288B active param model, which would still be by ~far (especially in terms of active) today the biggest oss model if it was ever released
@eliebakouch still, i would be really surprised if they dont start from the largest model and distill downwards
@eliebakouch ant/oai https://x.com/eliebakouch/status/2075938296190894433?s=46&t=Pi9ad3LY5VSI8lQ5oFejBQ
@nrehiew_ oh yeah, that would make sense
Positive users endorse the argument that distillation offers limited competitive edge for frontier models, while negative users call the claims overblown, overrated, or fearmongering by labs like Anthropic.
Based on 18 visible X reactions from 99 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@oneill_c Valuable datapoint 💯