Noam Brown, OpenAI o1 co-creator, urges benchmark developers to plot LLM performance against test-time compute

Noam Brown, OpenAI o1 co-creator, urges benchmark developers to plot LLM performance against test-time compute · Digg

Posts from X

Most Activity

VIEWS93.9K

Suhail@Suhail

I had not fully considered this possibility before. Interesting.

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d93.9K534303

BOOKMARKS387

Ethan Mollick@emollick

This is worth reading.

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d90.4K465387

LIKES826RETWEETS61REPLIES32

Noam Brown@polynoamial

We've known about LLM test-time compute scaling since @OpenAI o1. Yet 2 years later labs still report scalar evals for models; safety orgs are still surprised when a scaffold does better via 100x inference; and RSPs still ignore inference budget when deciding critical thresholds.

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d71K826294

Aaron Levie@levie

Great post. So much about model performance is a function of how much compute you’re doing at inference time. This means compute-normalized benchmarks is the only logical path forward.

And yet, the challenge is it’s a lot harder than it seems given it’s subjective how much compute to apply, which means models behave differently at different thresholds (simplistically, model X’s min thinking may beat model Y’s min thinking, but be reversed at high), and there are a near infinite set of thresholds you could choose to set.

But either way, moving more in this direction would be great for better understanding AI progress.

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d26K8880

Julian Schrittwieser@Mononofu

Plotting benchmark results with inference cost on the x-axis is absolutely the right thing to do, great writeup by @polynoamial !

I'm also excited to see that the new https://cognition.ai/blog/frontier-code has exactly such plots

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d18K9455

Anjney Midha@AnjneyMidha

this is a good proposal from @polynoamial

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d13.5K4545

Anastasios Nikolas Angelopoulos@ml_angelopoulos

"I believe the proper way to evaluate models is with a performance vs test-time compute plot, with either tokens, cost, or wall-clock time on the x-axis."

We can do this on Agent Arena data! Here's a plot showing net improvement vs tokens on 100K+ real agent workflows on @arena!

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d12.1K6921

Jasmine Wang@j_asminewang

This was one of my favorite talks at Recursive!

I like this recommendation: Preparedness Frameworks and Responsible Scaling Policies should explicitly account for inference compute [scaling] when determining whether a model crosses a safety threshold.

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d3.3K3213

Erik Brynjolfsson@erikbryn

How good is a model if you let it just keep doing more and more test-time compute?

Maybe the sky's the limit.

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d5.2K138

rohit@krishnanrohit

This is a great article. And identifying how the plateau changes with added inference for non verifiable tasks, like writing, would be extremely useful to know. I rather find a U shape already sometimes between thinking and pro so it's a useful area to note.

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d3.3K178

Chris@ChrissGPT

Noam,

“performance plateau even farther out. If this trend continues, which I fully expect, benchmark scores that don’t account for inference compute usage will become less informative each model release cycle.”

This still leaves room for me to ask if you think AGI (any median human task as good as a human for any duration) will be a function of increasing test time compute or if we will need to add more layers on the transformer stack or have a new architecture to reach agi?

22d2K252

elie@eliebakouch

> it may turn out that the only way to confidently evaluate misalignment in an AI agent at a 1-year horizon is to actually run the agent for a yea

this is a bit confusing imo, AI agent time is quite different from human time, 1 year horizon task is quite different from running the agent for 1y no?

you can probably find a hardware/parallelism config that optimizes speed for very long evals, or even tradeoff sequential test time compute with parallel test time compute? (but then it's a bit different i agree)

also output token is not perfect for things like autoresearch, a big portion of the time is actually spent in "tool call" which here are training runs

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

21d2.7K193

Eric Horvitz@erichorvitz

Important finding.

Noam Brown@polynoamial

http://x.com/i/article/2057694226981257216

22d1.8K45

Norman Mu@TheNormanMu

@polynoamial what do you make of the lack of scaling on Cognition's new benchmark? https://cognition.ai/blog/frontier-code

22d1.3K82

0xSero@0xSero

@polynoamial

22d1K43

0xSero@0xSero

@polynoamial

22d2.4K102

Chris@ChrissGPT

"Yet 2 years later labs still report scalar evals for models" OpenAI still does this? this is still important? Mythos is extremely token efficient and scales quite well ? I only care about specific benchmarks I helped design and assume the cost will fall orders of magnitudes

mythos isnt 100x the inference of the previous model and scores much higher on most benchmarks

21d1.3K121

Tahseen Rahman@Tahseen_Rahman

The capability jump in Fable is real on benchmarks, but production AI agents still hit diminishing returns past a certain context length. Most indie teams shipping paid tools right now get better results chaining specialized models than relying on one frontier release. Evaluation loops and user feedback loops matter more than the next model drop.

21d1483

Mert Gulsun@mert_gulsun

@polynoamial @0xSero Indeed, that’s why our benchmarks at @ArtificialAnlys build Pareto frontiers.

From the GPT-5.5 model card:

22d1.2K19

Przemek Chojecki | PC@prz_chojecki

@polynoamial Great article! We've just introduced ErdosBench to get multi-layered LLM benchmark on open math problems:

22d22821