Pseudonymous commentator Teortaxes positions a recent GLM release from Z.ai as closing in on Claude 3 Opus performance levels through efficient methods rather than massive data or compute outlays, with distillation floated as one possible route around traditional Chinese data constraints. Available details stay limited to coding-focused benchmarks on GLM-5 variants, and direct head-to-head verification against the specific 2024 Opus model is not established.
GLM blows a big hole in my thesis that the Chinese have a catastrophic disadvantage in high-quality data. It's too close to Opus. They did this without spending billions. I don't know how. Maybe distillation is all you need, like it served to bootstrap early assistants. Wild.
A reply from research engineer kalomaze flagged potential weaknesses in the value estimation approach used for the comparison, underscoring that benchmark framing can shift how close these models truly appear.
Posts note the GLM series relies on MoE designs and Huawei-chip training to keep costs low, yet no primary confirmation exists on exact training data volume or distillation steps behind the cited results.
Positive users praise Chinese labs for efficiently closing the gap to Claude 3 Opus via distillation and synthetic data, while negative users respond with insults against Americans and sarcastic dismissals of the claims.
No Digg Deeper questions have been answered for this story yet.
Anthropic is too addicted to big revenue to stop offering their API globally
Chinese labs are encouraged to do whatever it takes to catch up
The distillations will continue as long as the models improve.
GLM blows a big hole in my thesis that the Chinese have a catastrophic disadvantage in high-quality data. It's too close to Opus. They did this without spending billions. I don't know how. Maybe distillation is all you need, like it served to bootstrap early assistants. Wild.
@teortaxesTex *sigh* value estimation smell real?
GLM blows a big hole in my thesis that the Chinese have a catastrophic disadvantage in high-quality data. It's too close to Opus. They did this without spending billions. I don't know how. Maybe distillation is all you need, like it served to bootstrap early assistants. Wild.
@beffjezos I suspect that it's impossible to stop short of Fable-style lockdown Extraction of the reward profile delta seems to be absurdly more efficient than copying RL-d behavior with SFT capability lift. You can bootstrap with a few tens of thousands of innocuous interactions.
Anthropic is too addicted to big revenue to stop offering their API globally
Chinese labs are encouraged to do whatever it takes to catch up
The distillations will continue as long as the models improve.
@teortaxesTex Better RL environments for coding? you shouldn't really need better data at some point, if you can get the RL working well enough
also I wonder if this is sanctions helping them. more gaming gpus which are not as good at training but better for RL
GLM blows a big hole in my thesis that the Chinese have a catastrophic disadvantage in high-quality data. It's too close to Opus. They did this without spending billions. I don't know how. Maybe distillation is all you need, like it served to bootstrap early assistants. Wild.
@teortaxesTex @chiefofautism They have a program where you can paste a repo or search a repo and zai would create an entire wiki on it it was part of their MCPs/offering last year
I don’t see it on the internet anymore but I had a post about it that blew up @grok it was about pi as well
@teortaxesTex distillation is more powerful than ever in the llm as judge rl regime. a very chinese strategy, and they do it best. they let the westoids do the data legwork, exercise their "taste". meanwhile they engineer sota techniques to wring out every last drop of opus's soul.
@0xSero @chiefofautism zwiki?
@teortaxesTex @chiefofautism nevermind it's this
@teortaxesTex they didnt spend billions
as long as you completely ignore the hundreds billions of dollars in subsidies the chinese government has poured into AI, energy, vouchers, infra, etc
an example of this is a task that takes paragraphs, sorts sentences into some ~random order, asks the lm to reconstruct a valid one there's vastly many more wrong orders than correct orders, so pinning to the original order is sane - but the original order isn't uniquely determined by the input in a lot of cases the value estimator, in principle, learns a way to assign relative points that hedge as best as is possible given the incompressible constraints. that's probably the true value (heh) in explicit estimation; not some vague """variance reduction""" for """sample efficiency""" in the sense that the canon literature likes to invoke (else robotics people could just crank GRPO group size), but the smooth manifold geometry with inherently relative characteristics
@LLMJunky The long and short of what you're claiming is that your companies are trash and so is your government, so you deserve to lose
Beijing has 6 world class universities with more than 40k grads, post grads and phd students “EACH”. And that is Beijing alone. Z AI, MiniMax, ByteDance SEED etc are all based there. That cluster publishes worlds most books and research papers across all subjects.
I always find it laughable that they can not prepare datasets for pre and post training.
@TeleCat88 @teortaxesTex @grok how many people died in china as a result of starvation, political purges, repression, and executions during the same time period of the vietnam war
and then compare that to how many died in the vietnam war on both sides.
@teortaxesTex did you try it yourself or is it based on benchmarks
@Gusarich I am using it
@teortaxesTex @grok what happened in tiananmen square, how many people are estimated to have been murdered by the Chinese government
@kalomaze @teortaxesTex PPO won
@LLMJunky I didn't even get notifications for your subhuman blather You have to understand a few simple points. The CCP is the legitimate government of China, it has performed quite well over the last 40+ years, democracy is not the best possible system, you're simply brainwashed.
@LLMJunky > insect noises enough concern trolling, trash
@teortaxesTex The concept of copyright doesn’t exist. If anything they have the same data or better.