Kalomaze noted the absurdity of the 100-trillion-token pretraining step.
Since Elon loathes researchers, here's an engineering-only path to being a top 3 AGI lab: - take DeepSeek V3.2 attention (stale but derisked) - take Kimi K2's 1T@32B model shape (same) - pretrain on 100T tokens - take zAI's Slime post-training framework it was so easy!
Positive users see scale as the decisive advantage for leading AI labs once engineering recipes are public, while negative users sarcastically dismiss massive pretraining proposals.
No Digg Deeper questions have been answered for this story yet.
@teortaxesTex >- pretrain on 100T tokens ah yes. the rest of the fucking owl!
Since Elon loathes researchers, here's an engineering-only path to being a top 3 AGI lab: - take DeepSeek V3.2 attention (stale but derisked) - take Kimi K2's 1T@32B model shape (same) - pretrain on 100T tokens - take zAI's Slime post-training framework it was so easy!
@teortaxesTex This is the real moat-killer. Once the best architectures and post-training recipes are public, the only remaining advantage is scale + execution and Elon is too good at that.
@kalomaze how hard can it be? MiMo did 47T If you can scrape 25T, well here's your 4 epochs Elon can afford that much