Studying benchmarks after accuracy saturates
The arXiv description of “Life After Benchmark Saturation” argues that replacing benchmarks with harder versions once accuracy saturates privileges accuracy and misses opportunities for study. An author says “NeurIPS / Claude” rejected the paper.
TLDR
The arXiv description of “Life After Benchmark Saturation: A Case Study of CORE-Bench” says benchmarks are often retired and replaced with harder versions when their accuracy saturates. Its authors argue that this approach privileges accuracy and misses opportunities for study. An author says “NeurIPS / Claude” rejected the paper but maintains that it is an important contribution to evaluation science.
Combined views
15K
1 Source, first seen 1h ago
