Report
SolomonMg warns AI benchmarks are noisier than they look
Citing a NeurIPS paper, SolomonMg says prompts, judges and agent harnesses introduce overlooked noise into evaluations.
TLDR
SolomonMg highlights a new NeurIPS paper arguing that overlooked pipeline noise causes errors in AI benchmarking and annotation to be underestimated. The post says the problem worsens with more data and makes benchmark hacking easier. In response, srchvrs argues new statistical approaches may not be needed if random errors can be assumed independent.
Combined views
11.2K
6 Sources, first seen ago

Comments