Leading AI models reportedly nearly max out corrected physics benchmarks

@ZeyiAndyLiuNew paper: How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks Frontier models have tackled Navier–Stokes. People naturally wonder, how is their capability for physical sciences? According to today’s benchmarks, frontier models still struggle with physics (e.g. 47.3% with GPT5.6-sol on Humanity Last Exam - Physics component). But that picture did not match physicists' experience using these models. So we did a rigorous analysis of current benchmarks and re-graded their supposed failures with Yale physicists. We found that the problem was often the benchmark and not the model !! When corrected, the models almost saturate all benchmarks, including everyone’s favourite, Humanity Last Exam (physics-part) and Critpt. These failures in turn make AA less reliable as a measure of model’s true capability. For more details, results and analysis check out our preprint: https://arxiv.org/abs/2609.13009 and blog post: https://jsous.github.io/blogs/is-physics-dead/

@armancohanRT @ZeyiAndyLiu: New paper: How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Le…

@scaling01turns out most benchmarks are trash "When corrected, the models almost saturate all benchmarks, including everyone’s favourite, Humanity Last Exam (physics-part) and CritPt."

@ChrisGPTEver since Artificial Analysis pinned GPT-6 Astra below many others we all felt something was wrong, now we may have evidence. A new paper had Yale physicists manually re-grade the supposed failures of frontier models across major physics benchmarks, and found that a huge amount of the problem was actually the benchmark/grading itself, not the models. This is something @tszzl and many other lab employees note. Once the broken questions and grading were corrected, frontier models nearly saturated several of these benchmarks. GPT-5.6 Sol for example jumps from 47.3% to 78.7% on HLE-Physics mean@4, and hits 94.4% pass@4 on corrected CritPt. This is a pretty brutal indictment of the index and I hope they are able to look into these results!

@emollickThe state of public AI benchmarking is dire and is undermining our ability to understand how good AI is now. Most famous measures are maxed out, and, as this paper shows, the non-saturated benchmarks are riddled with so many errors that they vastly underestimate AI abilities.

@sanderstedRT @ChrisGPT: Ever since Artificial Analysis pinned GPT-6 Astra below many others we all felt something was wrong, now we may have evidence…

@sanderstedRT @ZeyiAndyLiu: New paper: How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Le…

@rohanpaul_aiNew Yale Univ + other top lab paper shows frontier models are already close to maxing out today’s closed-ended physics benchmarks. And many apparent failures come from bad questions, wrong reference answers, and brittle graders explain most audited physics failures, making benchmark quality the new bottleneck for measuring frontier models. The researchers had physicists re-check model failures across 6 popular physics benchmarks instead of trusting the original scores. In 250 rejected cases from 4 audited benchmark subsets, only 12 were actual model mistakes. The other 238 came from bad questions, wrong reference answers, or graders rejecting correct answers. After expert review, GPT-5.6-Sol’s measured HLE-Physics score rose from 47.3% to 78.7%. So a low physics benchmark score can badly underestimate what a frontier model can actually solve. But that does not mean these models can reliably do physics research: the authors’ agents still failed to fully solve any of the open theoretical-physics problems they tried. but it does mean that do not treat benchmark scores as clean ground truth anymore for AI's Physics capability.

@rohanpaul_ai– https://arxiv.org/abs/2609.13009 Title: "How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks"

@rohanpaul_aiRT @rohanpaul_ai: New Yale Univ + other top lab paper shows frontier models are already close to maxing out today’s closed-ended physics be…

Leading AI models reportedly nearly max out corrected physics benchmarks

TLDR

Combined views

Leading AI models reportedly nearly max out corrected physics benchmarks

10 Sources

Combined views

Sentiment

Summary

TLDR

Sentiment

Summary

Sentiment

Summary

Sentiment

Summary

10 Sources