I had not fully considered this possibility before. Interesting.

This is worth reading.

We've known about LLM test-time compute scaling since @OpenAI o1. Yet 2 years later labs still report scalar evals for models; safety orgs are still surprised when a scaffold does better via 100x inference; and RSPs still ignore inference budget when deciding critical thresholds.

Great post. So much about model performance is a function of how much compute you’re doing at inference time. This means compute-normalized benchmarks is the only logical path forward.

And yet, the challenge is it’s a lot harder than it seems given it’s subjective how much compute to apply, which means models behave differently at different thresholds (simplistically, model X’s min thinking may beat model Y’s min thinking, but be reversed at high), and there are a near infinite set of thresholds you could choose to set.

But either way, moving more in this direction would be great for better understanding AI progress.

Plotting benchmark results with inference cost on the x-axis is absolutely the right thing to do, great writeup by @polynoamial !

I'm also excited to see that the new https://cognition.ai/blog/frontier-code has exactly such plots

this is a good proposal from @polynoamial

"I believe the proper way to evaluate models is with a performance vs test-time compute plot, with either tokens, cost, or wall-clock time on the x-axis."

We can do this on Agent Arena data! Here's a plot showing net improvement vs tokens on 100K+ real agent workflows on @arena!

This was one of my favorite talks at Recursive!

I like this recommendation: Preparedness Frameworks and Responsible Scaling Policies should explicitly account for inference compute [scaling] when determining whether a model crosses a safety threshold.

How good is a model if you let it just keep doing more and more test-time compute?

Maybe the sky's the limit.

This is a great article. And identifying how the plateau changes with added inference for non verifiable tasks, like writing, would be extremely useful to know. I rather find a U shape already sometimes between thinking and pro so it's a useful area to note.

Noam,

“performance plateau even farther out. If this trend continues, which I fully expect, benchmark scores that don’t account for inference compute usage will become less informative each model release cycle.”

This still leaves room for me to ask if you think AGI (any median human task as good as a human for any duration) will be a function of increasing test time compute or if we will need to add more layers on the transformer stack or have a new architecture to reach agi?

> it may turn out that the only way to confidently evaluate misalignment in an AI agent at a 1-year horizon is to actually run the agent for a yea

this is a bit confusing imo, AI agent time is quite different from human time, 1 year horizon task is quite different from running the agent for 1y no?

you can probably find a hardware/parallelism config that optimizes speed for very long evals, or even tradeoff sequential test time compute with parallel test time compute? (but then it's a bit different i agree)

also output token is not perfect for things like autoresearch, a big portion of the time is actually spent in "tool call" which here are training runs

Important finding.

@polynoamial what do you make of the lack of scaling on Cognition's new benchmark? https://cognition.ai/blog/frontier-code

@polynoamial

@polynoamial

"Yet 2 years later labs still report scalar evals for models" OpenAI still does this? this is still important? Mythos is extremely token efficient and scales quite well ? I only care about specific benchmarks I helped design and assume the cost will fall orders of magnitudes

mythos isnt 100x the inference of the previous model and scores much higher on most benchmarks

The capability jump in Fable is real on benchmarks, but production AI agents still hit diminishing returns past a certain context length. Most indie teams shipping paid tools right now get better results chaining specialized models than relying on one frontier release. Evaluation loops and user feedback loops matter more than the next model drop.

@polynoamial @0xSero Indeed, that’s why our benchmarks at @ArtificialAnlys build Pareto frontiers.

From the GPT-5.5 model card:

@polynoamial Great article! We've just introduced ErdosBench to get multi-layered LLM benchmark on open math problems: