Commentary on X
@soldni @jwuphysics @jxmnop heartbreaking. i almost though microsoft had a heart... phi must die, huh? it is never too late. torrent the weights. you know you wanna
3:29 PM · Jul 15, 2026@andrew_n_carr And basically all recent web and code data is just Claude and GPT The distillation angle is so tiresome
2:57 PM · Jul 15, 2026@teortaxesTex They treat distilling like war crime or something which is so hilarious to me
3:28 PM · Jul 15, 2026View all
Pretraining a decent LLM from scratch requires a lot of _process knowledge_. Even if most of the value-add comes from post-training, knowing how to do everything from scratch both has a lot of positive externalities and provides a lot of resilience to the ecosystem. There are only a handful of players today (Ant, OAI, GDM) who we know can properly pre-train a model, even though in theory the ability to do this should be much more widespread. Another notch on this list is something to celebrate. Relying on the benefaction of open-weight model developers from other countries is not a robust long-term strategy.
4:26 PM · Jul 15, 2026> • Kimi distills • GLM distills • Qwen distills • Nemotron distills (Kimi & DeepSeek, which counts) you had one job…
3:22 PM · Jul 15, 2026