Frontier AI Models Score Nearly Identically on Coding Benchmark
Experts discuss non-monotonic performance curves and benchmark quirks for top models.
Entities: FrontierCode, Opus 5
A leaderboard screenshot for FrontierCode v1.1 shows two leading agentic coding systems at 53.4% and 53.5%. Commenters including Thomas Wolf, Jared Palmer, and Christopher Potts note that raising reasoning effort on Opus 5 reduces scores, unlike other models, and highlight effects from pure RL versus synthetic scaling. The discussion centers on why additional test-time compute sometimes hurts results and what this reveals about current evaluation limits.
This closely aligns with findings we report for Opus 4.6 in this post: https://bigspin.ai/resources/the-decline-of-token-level-purchasing-power
@Xinyu2ML the full graph makes this even weirder
Reactions from ranked influencers
10 postsoai went through a similar phase they just dont have people who can industrialize
@Xinyu2ML the full graph makes this even weirder
pure rl vs. synth/very offline test-time scaling
@Xinyu2ML the full graph makes this even weirder
Interestingly, as you increase Opus 5 reasoning effort beyond medium, its score on FrontierCode _decreases_ unlike Fable and Sol 🤔
@Xinyu2ML the full graph makes this even weirder
We found that Opus 5 actually scores lower on FrontierCode 1.1 in xhigh and max reasoning efforts. Our investigation revealed that the FrontierCode "scope" score was penalizing Opus 5 for making changes unrelated to the task.
Interesting non-monotonic success-effort curve for Opus 5 on FrontierCode
@Xinyu2ML the full graph makes this even weirder
(emergent property of *smarter models) i should check my posts before they go out