IRT Noise in ECI Leaderboards Urges Caution on AI Model Rankings
Reactions from ranked influencers
2 posts
not a critique of the ECI per se, thats just how IRT works. but it is important to know these things to accurately interpret the results. the error bars are also massive for all models, so saying that one model is strictly superior cause it got one or two more pts is silly
illustrative example: the software subset of the ECI comes with Mechanize's GBAEval, where GLM-5.2 scores 0, likely due to it being text-only (so it cannot verify its outputs). excluding this bench gives GLM-5.2 a huge jump. some of the top models don't have GBAEval scores
@teortaxesTex The problem with *all* IRT applications is that it’s relatively noisy. It is elegant, but you have to look at the error bars and take the results with two big scoops of salt. Adding or removing just one of the scores can have 1-2 pts of impact, which is massive