Noise in AI Benchmarks Undermines Tiny Model Score Debates
Reactions from ranked influencers
2 posts
@teortaxesTex My fav example is: SimpleQA-Verified is a bench which no one cares about anymore, we’ve moved on to web search agents. So it you include this bench’s scores for a modern model, they lose 0.5-2 pts. All within error range, but ppl on the TL debate over 156 vs 157
IRT is often used to reduce big eval suites to smaller sizes. Prerequisite for this are a lot of existing runs to find the samples You’d be shocked to see how many papers perform as well as a random selection of samples
@teortaxesTex The problem with *all* IRT applications is that it’s relatively noisy. It is elegant, but you have to look at the error bars and take the results with two big scoops of salt. Adding or removing just one of the scores can have 1-2 pts of impact, which is massive