China Develops 100x More LLM Training Talent Than US Through Internships
Proprietary methodologies force researchers to redundantly rediscover training techniques.
Entities: Guohao Li, Igor Carron, Taco Cohen
A post from @TacoCohen is circulating a claim about China developing far more LLM training talent than the U.S. through internships, but the visible public evidence is still narrow. For now, the safest read is to stick to what the post says, who put it in front of the conversation, and what remains unverified before anyone treats the claim as settled.
Combined views
47K
9 posts, first seen 1d ago
Reactions from ranked influencers
9 posts
Pour les français qui me lisent: ceci est la seule recette qui marche. cela a marcher pour Bloom, pour @LightOnIO . pas un machin comme promethee.
"most of the work done in the china labs is carried by interns. i met brilliant undergrad and graduate interns who deeply understand the model training details, and they are 100x more open to share. that means the talent that knows how to train llms in china is 100x greater in number than the talent in the us, and it is growing in contrast, the us ai ecosystem is too closed. frontier labs do not hire interns. i know brilliant phd students at stanford, berkeley, and so on. they struggle to get an internship and the compute to train a properly sized model."
Don't kill the golden goose that keeps the AI party alive! Open ecosystems generate the very ideas that prevent progress from stalling. Frontier AI develops at the pace of distributed SGD across ideas, people, and constraints. The era of Machine Learning and Deep Learning (2014-2022) thrived on broad openness in sharing and publishing. Partnership between labs, industry and universities got us multiple breakthroughs including the what we now call frontier AI and secured the US AI leadership. In contrast, closed ecosystems inherently slow down knowledge diffusion by reducing the number of independent ideas attempted. ArXiv culture is precisely what enabled fast ML iteration; a similar openness is why hardware ecosystem development in China moves so quickly, and why model development thrives in open ecosystems today. AI is still evolving rapidly, with multiple open frontiers: physical, recursive/continual learning, power efficiency, data efficiency, among others. While AI is transformational, attempts to hoard knowledge in-house is an unfortunate local minimum used merely to justify value to capital. Enabling the broader ecosystem to participate moves the technology forward faster. While innovation on product and GTM layer is vital and rather expected, timing and ordering matters! Closing the doors too early in the cycle will slow innovation on the underlying tech layer and effectively stall product quality. The attention, resources, and capital coming from outside the core AI world are ephemeral. If progress stalls or becomes more expensive than it already is, the spotlight will move on. Don't kill the golden goose. Humanity needs AI.
why can the china labs build glm-5.2, kimi k3, and many more to come? it is because of the openness. not just the open weights but the whole ecosystem. most of the work done in the china labs is carried by interns. i met brilliant undergrad and graduate interns who deeply understand the model training details, and they are 100x more open to share. that means the talent that knows how to train llms in china is 100x greater in number than the talent in the us, and it is growing in contrast, the us ai ecosystem is too closed. frontier labs do not hire interns. i know brilliant phd students at stanford, berkeley, and so on. they struggle to get an internship and the compute to train a properly sized model. most of the secret recipes are locked away by a very small group of privileged researchers it is not about china or the us. it is about open and closed science. the fact is that every average cs student can learn how to train an llm. they just need the opportunity. labs should be more open and hire more interns, like how deepmind and fair did in the pre-llm era
@natolambert Wondering if the money will stop happening if progress stops. And progress is dependent of new ideas from th ecosystem. Otherwise even large closed orgs suffer from context rot
Don't kill the golden goose that keeps the AI party alive! Open ecosystems generate the core ideas that prevent progress from stalling. Frontier AI develops at the pace of distributed SGD across ideas, people, and constraints. The era of Machine Learning and Deep Learning (2014-2022) thrived on broad openness in sharing and publishing. Partnership between labs, industry and universities got us multiple breakthroughs including the what we now call frontier AI and secured the US AI leadership. In contrast, closed ecosystems inherently slow down knowledge diffusion by reducing the number of independent ideas attempted. ArXiv culture is precisely what enabled fast ML iteration; a similar openness is why hardware ecosystem development in China moves so quickly, and why model development thrives in open ecosystems today. AI is still evolving rapidly, with multiple open frontiers: physical, recursive/continual learning, power efficiency, data efficiency, among others. While AI is transformational, attempts to hoard knowledge in-house is an unfortunate local minimum used merely to justify value to capital. Enabling the broader ecosystem to participate moves the technology forward faster. While innovation on product and GTM layer is vital and rather expected, timing and ordering matters! Closing the doors too early in the cycle will slow innovation on the underlying tech layer and effectively stall product quality. The attention, resources, and capital coming from outside the core AI world are ephemeral. If progress stalls or becomes more expensive than it already is, the spotlight will move on. Don't kill the golden goose. Humanity needs AI.
Agree with this: US labs are underestimating the power and vitality of open research. This is one of my favorite essays and I think it rings true on the macro level not just the micro level. https://www.cs.virginia.edu/~robins/YouAndYourResearch.html
why can the china labs build glm-5.2, kimi k3, and many more to come? it is because of the openness. not just the open weights but the whole ecosystem. most of the work done in the china labs is carried by interns. i met brilliant undergrad and graduate interns who deeply understand the model training details, and they are 100x more open to share. that means the talent that knows how to train llms in china is 100x greater in number than the talent in the us, and it is growing in contrast, the us ai ecosystem is too closed. frontier labs do not hire interns. i know brilliant phd students at stanford, berkeley, and so on. they struggle to get an internship and the compute to train a properly sized model. most of the secret recipes are locked away by a very small group of privileged researchers it is not about china or the us. it is about open and closed science. the fact is that every average cs student can learn how to train an llm. they just need the opportunity. labs should be more open and hire more interns, like how deepmind and fair did in the pre-llm era
Don't kill the golden goose that keeps the AI party alive! Open ecosystems generate the core ideas that prevent progress from stalling. Frontier AI develops at the pace of distributed SGD across ideas, people, and constraints. The era of Machine Learning and Deep Learning (2014-2022) thrived on broad openness in sharing and publishing. Partnership between labs, industry and universities got us multiple breakthroughs including the what we now call frontier AI and secured the US AI leadership. In contrast, closed ecosystems inherently slow down knowledge diffusion by reducing the number of independent ideas attempted. ArXiv culture is precisely what enabled fast ML iteration; a similar openness is why hardware ecosystem development in China moves so quickly, and why model development thrives in open ecosystems today. AI is still evolving rapidly, with multiple open frontiers: physical, recursive/continual learning, power efficiency, data efficiency, among others. While AI is transformational, attempts to hoard knowledge in-house is an unfortunate local minimum used merely to justify value to capital. Enabling the broader ecosystem to participate moves the technology forward faster. While innovation on product and GTM layer is vital and rather expected, timing and ordering matters! Closing the doors too early in the cycle will slow innovation on the underlying tech layer and effectively stall product quality. The attention, resources, and capital coming from outside the core AI world are ephemeral. If progress stalls or becomes more expensive than it already is, the spotlight will move on. Don't kill the golden goose. Humanity needs AI.
why can the china labs build glm-5.2, kimi k3, and many more to come? it is because of the openness. not just the open weights but the whole ecosystem. most of the work done in the china labs is carried by interns. i met brilliant undergrad and graduate interns who deeply understand the model training details, and they are 100x more open to share. that means the talent that knows how to train llms in china is 100x greater in number than the talent in the us, and it is growing in contrast, the us ai ecosystem is too closed. frontier labs do not hire interns. i know brilliant phd students at stanford, berkeley, and so on. they struggle to get an internship and the compute to train a properly sized model. most of the secret recipes are locked away by a very small group of privileged researchers it is not about china or the us. it is about open and closed science. the fact is that every average cs student can learn how to train an llm. they just need the opportunity. labs should be more open and hire more interns, like how deepmind and fair did in the pre-llm era
This is spot on. There are a few tricks that every US lab has to discover independently or pay huge sums to the few insiders who know because they did it before. It’s much more efficient to have an open research ecosystem. The US labs have trapped themselves in a bad equilibrium.
why can the china labs build glm-5.2, kimi k3, and many more to come? it is because of the openness. not just the open weights but the whole ecosystem. most of the work done in the china labs is carried by interns. i met brilliant undergrad and graduate interns who deeply understand the model training details, and they are 100x more open to share. that means the talent that knows how to train llms in china is 100x greater in number than the talent in the us, and it is growing in contrast, the us ai ecosystem is too closed. frontier labs do not hire interns. i know brilliant phd students at stanford, berkeley, and so on. they struggle to get an internship and the compute to train a properly sized model. most of the secret recipes are locked away by a very small group of privileged researchers it is not about china or the us. it is about open and closed science. the fact is that every average cs student can learn how to train an llm. they just need the opportunity. labs should be more open and hire more interns, like how deepmind and fair did in the pre-llm era