OpenAI says SWE-Bench Pro has hit a 70% noise ceiling and no longer reliably measures frontier model capabilities
The company says a popular public coding benchmark is now too saturated to cleanly separate top models.
426521013752.7K
Story Brief
OpenAI said on X that it audited SWE-Bench Pro, called it "one of the most widely used AI coding benchmarks," and concluded the eval no longer reliably measures frontier coding capability because it has hit a roughly 70% noise ceiling. The company also said it is retracting its earlier recommendation that researchers use SWE-Bench Pro as a leading coding eval.
test models on your own benchmarks, public ones become benchmaxxed soooooo fast
Alex PeysTECHvia X
