Plotting benchmark results with inference cost on the x-axis is absolutely the right thing to do, great writeup by @polynoamial !
I'm also excited to see that the new https://cognition.ai/blog/frontier-code has exactly such plots
http://x.com/i/article/2057694226981257216
Users praise AI benchmark plots with inference costs on the X-axis as validating and more sensible than cost-free benchmarks that amount to mere flexing.
The only addition I'd make is that we should make sure to use a *log* x-axis, since most of the trends involved in test-time compute are logarithmic in compute.
Plotting benchmark results with inference cost on the x-axis is absolutely the right thing to do, great writeup by @polynoamial !
I'm also excited to see that the new https://cognition.ai/blog/frontier-code has exactly such plots
@Mononofu @polynoamial cost x-axis makes more sense than people realize
benchmarks without cost context are just flexing
@Mononofu @polynoamial nothing more validating than seeing ur approach independently confirmed on a new platform
No Digg Deeper questions have been answered for this story yet.
Plotting benchmark results with inference cost on the x-axis is absolutely the right thing to do, great writeup by @polynoamial !
I'm also excited to see that the new https://cognition.ai/blog/frontier-code has exactly such plots
http://x.com/i/article/2057694226981257216
