Reactions from ranked influencers
5 posts
they are coping so hard that they are already making up numbers to discredit ECI
@teortaxesTex My fav example is: SimpleQA-Verified is a bench which no one cares about anymore, we’ve moved on to web search agents. So it you include this bench’s scores for a modern model, they lose 0.5-2 pts. All within error range, but ppl on the TL debate over 156 vs 157
@scaling01 I called the SimpleQA impact for open models such as GPT-OSS end of last year in private to them like it’s a fundamental thing of IRT, dunno what you are on about
illustrative example: the software subset of the ECI comes with Mechanize's GBAEval, where GLM-5.2 scores 0, likely due to it being text-only (so it cannot verify its outputs). excluding this bench gives GLM-5.2 a huge jump. some of the top models don't have GBAEval scores https://twitter.com/xeophon/status/2078704457425076439
@scaling01 Fair that my impact for modern models was overstated, good that you did the analysis.
Mean change: +0.21 ECI 27 of 30 change by less than 0.5 ECI
@scaling01 he's been showing the flaws in ECI for months and actively engages with how to improve it. A related post: https://www.interconnects.ai/p/latest-open-artifacts-21-open-model