Håvard Ihle's agentic WeirdML benchmark tests push Claude Opus 4.7 and GPT-5.5 to 90% accuracy

OK, this somewhat rescues Opus. I obviously want to see the trend for open models

babe wake up they saturated weirdML

@htihle amazing!! also, "fun" exploit (modern evals really is just patching all the possible exploits...)

curious: did a model reach your 20M token limit?

@htihle i like turn gating a bit more (vibes-based), followed by a token limit. worst is wall-clock time, it is just bad

and for infinite money, i like a no-regression / loop detection setup, i.e., a model may run forever unless it does not improve its output for X iterations

@xeophon Very rarely, I think it happened like 2 times or so in the whole run (two runs per 17 tasks for two models), mostly they used less than 10M, quite a few between 10 and 20M. I think I'll use token gating or something like that in the future.

@xeophon Thanks for the input, I'll test out both token and turn gating. I'll also probably go higher than 20M for WeirdML v3, but we'll see.

Here is the website with more details: https://htihle.github.io/agentic_weirdml.html

@htihle they obviously all have their drawbacks, but i feel like models optimize more for parallel tool calls these days vs. raw per-dollar token efficiency. maybe i can have a data-based opinion after a ton more runs on hosted evals + swe benches :)

@htihle Would be great to see how Gemma4 31B performs in this as that’s the sota for consumer gpus…