Training an 800-billion parameter model requires 50,000 GB300 GPUs.
No Digg Deeper questions have been answered for this story yet.
Looks like Mythos is 800B Active params (the teacher model) not Fable 5/Mythos 5
LOTS OF ALPHA FROM WENFENG HERE He expects he won't be able to spend >20B RMB on compute in 2026 «The smartest people—maybe less than 50% stay in China» «The largest current model activates ≈800B parameters» «I'd need about 50K GB300s, or 200K Huawei 950s» [to train that]
Liang Wenfeng believes that the comprehensive gap in AI between China and the US is 12-18 months, just like Kai-Fu Lee says and Dario hopes; and can be shrunk to 3, with surpassing in some few key areas. @scaling01 @zephyr_z9
He thinks Huawei depreciation timeline is 3 years, 950s are too power-hungry more subsidies needed!
«16,000 Huawei 950 GPUs only equal 4,000 B-series GPUs. So it's not a large quantity—not very significant. Not enough to train a next-generation model—only enough to train our current generation. I don't really believe that five years from now we'll still be stuck on capacity. Now we're definitely stuck on capacity—this year, next year, the year after, I think we might still be stuck on capacity—but after five years, maybe not; I'm fairly optimistic» Is 5 years too long? Perhaps.
Liang Wenfeng believes that the comprehensive gap in AI between China and the US is 12-18 months, just like Kai-Fu Lee says and Dario hopes; and can be shrunk to 3, with surpassing in some few key areas. @scaling01 @zephyr_z9
«So we simply won't consider competing with the US at that scale now… when we have more resources later, we'll push to 150B, 156B, or 250B activation scale.» «You could force-train a model that big, but you couldn't do sufficient research» What are those models?
LOTS OF ALPHA FROM WENFENG HERE He expects he won't be able to spend >20B RMB on compute in 2026 «The smartest people—maybe less than 50% stay in China» «The largest current model activates ≈800B parameters» «I'd need about 50K GB300s, or 200K Huawei 950s» [to train that]
@scaling01 @zephyr_z9 It's probably grating to him that K3 or even GLM 5.2 are already useful for self-development. But again: "the first goal of the models we build isn't that users find them good to use, but that we find them good to use ourselves" explains a lot, really
Liang Wenfeng believes that the comprehensive gap in AI between China and the US is 12-18 months, just like Kai-Fu Lee says and Dario hopes; and can be shrunk to 3, with surpassing in some few key areas. @scaling01 @zephyr_z9
«When the global AI division of labor takes shape, the role Chinese companies most likely play is still the largest producer. Common sense says our production capacity is largest—including chips; our chip capacity may be largest, our electricity most abundant—so for AI we'll most likely end up as one of the three powers.» Which three?
«So we simply won't consider competing with the US at that scale now… when we have more resources later, we'll push to 150B, 156B, or 250B activation scale.» «You could force-train a model that big, but you couldn't do sufficient research» What are those models?
@scaling01 @zephyr_z9 «DeepSeek might already be at net profit» ofc that's because he cannot spend everything on GPUs (they don't exist)
@scaling01 @zephyr_z9 Wenfeng: «China has started to do high quality data annotation only in the last half year» I've been talking of this. Reduction in jaggedness with 5.2 and K3 are emblematic. (Distillation covers the rest)
Next generation DeepSeek must have continual learning. You see this objective in V4's paper, and you see it in hiring. Before they get there, they'll just be iterating on bang-for-the-buck.
Some explicit ideological rejection of American way. «America will meet challenges, and in the future it may meet China's challenge, because Chinese people are willing to take less and can provide you this service.»
V4's updated version should have native multimodality. V4 is as big as he could train at the moment. Suggests that either Kimi has more compute, or it has in his view made an error training something beyond their inference means.
On domestic compute: bullish within a year. «Domestic AI chips have no problems in hardware or ecosystem—the only problem is insufficient production capacity» «Previously, domestic GPU adaptation had a problem called poor ecosystem… The moat of NVIDIA's CUDA is being rapidly dismantled, for probably three reasons. - with AI, building this ecosystem is much easier than before, because AI can write code. - second, some new technologies. For example, our company produced a technology called TileLang—a high-level language. Using this high-level language to write CUDA operators, you can quickly rewrite NVIDIA's entire ecosystem, and combined with AI, there seem to be no obstacles. - Another point: because CUDA—NVIDIA evolved from gaming GPUs, so in many places the gaming GPU design and settings carried through»
950s vs Nvidia: «when V3 trained, it still used NVIDIA GPUs, but no longer used NVIDIA's ecosystem… As long as I redo this whole process on Huawei GPUs, it's done. I think this might be a historic mission» «Huawei 950 supernode can fully substitute for NVIDIA's GB200 and GB300 in performance and price» «four Huawei GPUs equal one NVIDIA GPU, and it's two years behind… So our chip gap with the US, I believe, will no longer exist in ecosystems, but in chips it's four-fold plus two years.»
Interesting contradiction with newer (and some older) reporting: «I hope not to make chips». But aren't you making chips already? I guess the funding round success + compute scarcity gave him more sense of urgency.
on costs, and again, the peril of trying to eat a large chunk of global GDP «We also have many algorithmic methods—costs can go lower still. «the lower the cost, the bigger the model I can train, the bigger the model I can afford»
more on corporate culture
«16,000 Huawei 950 GPUs only equal 4,000 B-series GPUs. So it's not a large quantity—not very significant. Not enough to train a next-generation model—only enough to train our current generation. I don't really believe that five years from now we'll still be stuck on capacity. Now we're definitely stuck on capacity—this year, next year, the year after, I think we might still be stuck on capacity—but after five years, maybe not; I'm fairly optimistic» Is 5 years too long? Perhaps.
Liang Wenfeng believes that the comprehensive gap in AI between China and the US is 12-18 months, just like Kai-Fu Lee says and Dario hopes; and can be shrunk to 3, with surpassing in some few key areas. @scaling01 @zephyr_z9
On Huawei: «we participate deeply in Huawei's ecosystem» «Huawei gives us capacity for about 16,000 GPUs; internet giants might get over a 100K… but this may already be all the capacity Huawei has.» «So we can't count on training the next bigger model on Huawei, or training models with several hundred B activated parameters… But next year or the year after, there might be a chance.»