Disabling thinking dropped Claude's accuracy to 70.5%.
This is such a zany pattern if you think about it
Claude Opus 4.8 (xhigh) scores 82.9% on WeirdML, right behind GPT 5.5.
We now also (unlike 4.7) see a clear scaling with output token use: - no thinking: 2.4k tokens, 70.5% - medium: 4.3k, 76.0% - xhigh: 12.5k, 82.9%
Positive users praise Claude Opus 4.8 for nearly matching GPT-5.5 on WeirdML with far fewer lines of code due to lower maintenance burdens, while negative users dismiss the results as insignificant or insult those promoting them.
No Digg Deeper questions have been answered for this story yet.
Opus 4.8-xhigh scores minimally lower than GPT-5.5-xhigh, but is absolutely simplicity-maxxing
129 LOC vs 517 LOC
I know which one I would pick
Claude Opus 4.8 (xhigh) scores 82.9% on WeirdML, right behind GPT 5.5.
We now also (unlike 4.7) see a clear scaling with output token use: - no thinking: 2.4k tokens, 70.5% - medium: 4.3k, 76.0% - xhigh: 12.5k, 82.9%
The big story here is that GPT 5.5 (high/xhigh) outperforms claude-opus-4.8 (max/xhigh) by 20.7% succeeding on 12 additional tasks!
More impressive: GPT is roughly half the cost and twice as fast.
OpenAI is back in the game. Overall, this competition is healthy for the industry. I'd love to see a third player rise to the top of the leaderboard!
Opus 4.8 is now on DeepSWE.
On the default high thinking effort, it scores 6% higher than Opus 4.7 xhigh, while also lowering average cost per task.
In WeirdML we see opus models increasingly use submissions to just explore the data, without actually trying to solve the problem (no predictions for the test set).
It seems like, with no or low thinking, at least for some tasks, the prior for Opus to just explore the data overrules the system instructions saying that the score on every submission counts and will be compared to other models.
Opus does not feel the rush to maximize the score fast, it simply tries to understand the data. While this kind of attitude is great in Claude-Code, it costs Opus a lot on an eval like WeirdML, where getting the most out of every submission is important.
Exploring the data is definitely crucial in WeirdML, and GPT and Gemini also do this (prints out a bunch of info, tests hypotheses about the data etc). They just also make their best effort to solve the task at every iteration (which means that they both get more info from each submission and also get valid scores etc).
Opus, somehow, does not feel the same urgency when in this eval. Not exactly sure what this means, and more thinking does make Opus use the submissions more effectively, but it's at least an interesting finding.
Claude Opus 4.8 (xhigh) scores 82.9% on WeirdML, right behind GPT 5.5.
We now also (unlike 4.7) see a clear scaling with output token use: - no thinking: 2.4k tokens, 70.5% - medium: 4.3k, 76.0% - xhigh: 12.5k, 82.9%
@scaling01 how to make your model's code concise and beautiful in 2 easy steps
1. train on my codebases
2. done
Opus 4.8-xhigh scores minimally lower than GPT-5.5-xhigh, but is absolutely simplicity-maxxing
129 LOC vs 517 LOC
I know which one I would pick
@scaling01 Same. A 2% benchmark gap means nothing if I have to maintain 517 lines instead of 129. Less code is less surface area to debug later. The model that writes tighter is the one I actually want in a real codebase.
@scaling01 depends if the 400 extra lines are error handling or the model just being verbose. score doesn't tell you which
@scaling01 which one?
@scaling01 i wonder if Opus 4.8 Max tops the benchmark.
@scaling01 and that is true, opus code is lot leaner than gpt 5.5, both work almost all of the times
@scaling01 129 vs 517 LOC is a real signal. does the thinking token count also diverge at xhigh, or does opus reach 82.9% with fewer reasoning tokens than gpt-5.5 uses?
@scaling01 same task, Opus makes the surgical edit, Codex rewrites half the file around it. finally a number for it.
@scaling01 129 LOC flexing while the model does all the heavy lifting
respect the math, hate the disrespect to my fingers
@scaling01 517 LOC to squeak out a marginal gain is not a flex it's a smell. Complexity compounds. In 6 months that's the codebase nobody wants to touch. Simplicity-maxxing isn't a consolation prize; it's the right call.
@scaling01 129 vs 517 LOC for nearly the same score is the whole argument honestly. Less code to maintain wins
@VictorTaelin @scaling01 why are you not pushing your comprehensive datasets to https://hub.harborframework.com/ which is used by terminal-bench. Many llms train on that.
@scaling01 That LOC delta is the real story — 75% less code for near-parity output suggests the architectural efficiency gap is widening, not shrinking. Curious if that simplicity advantage holds at scale or breaks under edge cases.
@scaling01 imo those 388 lines aren't missing, they're just in your future error logs
@scaling01 what's wrong with u? Can u see their output tokens? Or u just like A\? Marketing account
@VictorTaelin @scaling01 The goat
@scaling01 all this for a lego set