OpenAI withdraws endorsement of SWE-Bench Pro after audit finds 30% of its coding tasks are broken
The benchmark's 70% noise ceiling prevents meaningful frontier model evaluation.
Combined views
404.3K
30 Sources, first seen ago
Sources
FB 
Florian Brand@xeophonopenai looked at the data and it stared right back :( https://twitter.com/OpenAI/status/2074958149426241894
- likes: 59
- replies: 0
- bookmarks: 2
- reposts: 1
TR Talia Ringer 🕊🪬@TaliaRingerHow does this keep happening with AI benchmarks? Why don't these audits happen immediately? https://twitter.com/OpenAI/status/2074972179385720836
- likes: 50
- replies: 12
- bookmarks: 13
- reposts: 2
GN gavin leech (Non-Reasoning)@gleech- likes: 9
- replies: 1
- bookmarks: 1
- reposts: 0
FI finbarr@finbarrtimbersIt’s funny how common this is. Most, if not all, of the benchmarks I engage with have issues exactly like this. https://twitter.com/openai/status/2074972179385720836
- likes: 51
- replies: 3
- bookmarks: 5
- reposts: 2
JY John Yang@jyangballinKabir did a really fantastic job with this. Hence why SWE-bench Multilingual shows up on Anthropic Cursor/Grok, Cognition, GLM model cards consistently. It's a good benchmark. https://twitter.com/plodq/status/2075232370458800452
- likes: 18
- replies: 1
- bookmarks: 3
- reposts: 3
AP alex peysakhovich@alex_peystest models on your own benchmarks, public ones become benchmaxxed soooooo fast https://twitter.com/OpenAI/status/2074958149426241894
- likes: 2
- replies: 0
- bookmarks: 0
- reposts: 0