Epoch AI launches benchmark reviews, labels nine of 15 flawed

Epoch AI announced an initiative to audit AI benchmarks, classifying its first 15 as four verified, nine flawed and two with insufficient information for a review. A commentator defending the reviews against criticism argues that headline scores can hide problems with grading, task realism and model settings. They recommend inspecting failed examples before trusting a benchmark.