Research engineer Florian Brand warns METR benchmarks suffer from massive error bars
Hear me: People used to soyface about novel coding evals, where Chyna/open models were not just behind but garbage. GLM covered most of that gap. Now we look at combined metrics like ECI, or "pure reasoning" like ARC. I predict this, too, will prove to be surprisingly fragile.
"omg omg GLM-5.2 is beating fable. china is catching up"
chill out and listen to Lisan: > slightly ahead of Opus 4.5 > behind GPT-5.2, Gemini 3 Pro and Opus 4.6
Users dismissed the GLM-5.2 benchmark results versus GPT-5.2 and Claude Opus as mere Anthropic fanboyism.
No Digg Deeper questions have been answered for this story yet.
@teortaxesTex lol, a *lot* o the actual scores of GLM-5.2 are missing. no wonder its ECI is in the gutter when the scores where its (close to) SOTA are left out.
the GBAEval score from @MechanizeWork is also sus
cc @Jsevillamol @AlexBarry4
@teortaxesTex You’d think after one year of METR error bars so wide you can fit the whole AUM of Leopold in there, people would finally understand what they mean but alas
For all of Dario's fearmongering, for how seriously the US is taking the "AGI race", you can tell it's moslty a race between OpenAI and Anthropic. Evaluations for frontier Chinese open weights take weeks-months, if they happen at all. China is not a factor outside rhetoric.
@teortaxesTex lol, a *lot* o the actual scores of GLM-5.2 are missing. no wonder its ECI is in the gutter when the scores where its (close to) SOTA are left out.
the GBAEval score from @MechanizeWork is also sus
cc @Jsevillamol @AlexBarry4
lisan's (good) list of evals I welcome you to check how GLM 5.2 stacks up. Before code, it was math. The dirty secret is that American frontier really is frontier, and it's not just compute, they expand to new domains earlier. But if it's not compute…
the "narrow capability gap" in question
let's put this to rest please I can't hear the coping anymore
@teortaxesTex You’d think after one year of METR error bars so wide you can fit the whole AUM of Leopold in there, people would finally understand what they mean but alas
Hear me: People used to soyface about novel coding evals, where Chyna/open models were not just behind but garbage. GLM covered most of that gap. Now we look at combined metrics like ECI, or "pure reasoning" like ARC. I predict this, too, will prove to be surprisingly fragile.
@teortaxesTex @MechanizeWork @Jsevillamol @AlexBarry4 They don’t, ECI uses IRT. You need a handful of scores to calculate an aggregate and don’t need the same evals for all models, which is why IRT has bigger error bars
Problem is that the selection somewhat matters, so missing the good ones depresses scores
@xeophon @MechanizeWork @Jsevillamol @AlexBarry4 Wait how do they account for values of missing evals?
@xeophon @MechanizeWork @Jsevillamol @AlexBarry4 Wait how do they account for values of missing evals?
@teortaxesTex lol, a *lot* o the actual scores of GLM-5.2 are missing. no wonder its ECI is in the gutter when the scores where its (close to) SOTA are left out.
the GBAEval score from @MechanizeWork is also sus
cc @Jsevillamol @AlexBarry4
@teortaxesTex >People It's just that Anthropic fanboy