A proposal to make AI benchmark rankings more reliable

Quanta Magazine reports that computer scientist Moritz Hardt questions what an AI model’s absolute benchmark score tells us: a slightly different test set can produce a very different number. He says rankings between models have historically been more stable, but that advantage may not hold for large language models trained to different degrees for particular benchmarks. Hardt proposes preparing models on benchmark-relevant data before testing so the rankings better reflect their relative performance.