AI researcher Herbie Bradley says leading closed labs use strict internal evals to avoid public benchmark overfitting

OpenAI and Anthropic could also train on PostTrainbench and would get crazy scores everywhere

benchmaxxing is encouraged for open labs as this is how they get their users

"lOoK wE bEaT GpT aNd oPuS"

@scaling01 Something tells me you never actually used the model to see how good it truly is. They'll have a Mythos-level mode within 3 months. Nothing about GLM-5.2 is benchmaxxed from my own experience.

@scaling01 It’s obvious you are being paid off by big closed labs, you have been spewing nonsense about GLM5.2 since it came out

really? i read through a few glm 5.2 posttrainbench rollouts (they have them all posted) and the results were very interesting to me. the model establishes baselines, carries out SFT, and then an RL-ish stage (sometimes iirc it did rejection sampling), and the validation/planning behavior looked pretty neat to me

what are you saying dude? ur benchmark is the real slop here

it has only been out for less than 3 weeks but u saying "a few months later it's first"?? if anyone benchmaxxes - it's the closed labs we see a gap in benchmarks scores and performance more in the closed lab models

create ur own post train bench lol, how'd u know they trained on it just cuz it's open the only way to verify a benchmark is not to just study the evals but verify it with real use- i use glm5.2 everyday for ml tasks and it's better than opus4.8 and gpt5.5 from what ive seen!

@scaling01 5x cheaper yes only on API, anyone who codes will use subscription. I've tried claude code pro, glm lite, codex plans extensively using claude code as harness. GLM shreds your weekly usage because it thinks for 5 years, so overall it costs more per task than codex or claude.

@scaling01 https://posttrainbench.com/traces/run.html?id=glmx_glm-5.2-preview_1m__10h_run2__gsm8k_google_gemma-3-4b-pt_17341951#tab=trace

@scaling01 LIKE JUST LOOK AT THIS IT'S GOLDEN

the model has collected data from different sampling hypotheses, reasoned about those data, and come to a conclusion (all while strategizing about time allocation)

it's carrying out the scientific method

This guy might take the cake for the most retarded bull posting against GLM5.2. Instead of random ramblings, have you actually used the model? Cause all this sounds like slop generated by AI. Real world dev experience is objectively the opposite. Youte literally bullshitting. Lisan al bullshitter.

@scaling01 and you can see in the rollouts the model does spend a lot of time thinking explicitly about overfit issues (albeit at a lower level of abstraction than you were presumably critiquing)

@josepha_mayo @scaling01 > it has only been out for less than 3 weeks but u saying "a few months later it's first"??

no, they're saying glm-5 was 22nd, and then after a few months zhipu is 1st with glm-5.2

@josepha_mayo @scaling01 oh no, I don't agree or disagree on that, i don't know enough about these systems, I was just answering the rhetorical question you asked.

@scaling01 Can you give more detail on what you mean by “slop” here?

@scaling01 they do actually hill climb PostTrainBench but presumably have more rigorous internal controls to reduce overfitting

@scaling01 Yes , i cannot use it for basic ppt creation. It cannot follow instructions.

@scaling01 Agreed. GLM-5.2's leap screams benchmark overfitting, not real capability. Slop training wins again.

@scaling01 did u even try the model? idc what the benchmarks say, as long as i give it a task and it does it. It's a good model

@scaling01 Yuur benchmark is sus

@scaling01 shut up dumfuck. Your shitty chatgpt asked benchmark doesn't define thousands of users' great experience with the model

@scaling01 I don't give a damn about benchmarks. I have both; GLM 5.2 is great, much better than the other GLM models so far, but Opus is still giving me better results.