Frontier AI labs spend $10 billion to $15 billion each on training data as high-quality tasks reach $20,000

> "Really good long horizon tasks go up to $20,000 each. A complete browser-use version of SAP was rumored at $500,000." I think this points to market inefficiency

It's been really hard for METR to find vendors that can sell us long-horizon hard tasks that we can actually use.

There's 2 approaches to train+eval data acquisition: A. Finding it in the real world B. Paying someone to produce it

In the coding domain, you can see SWE-bench as the first category and TerminalBench as the second.

In coding, it might soon be impossible to do B. 🧵

A good benchmark provides a north star to guide & measure AI development. A great benchmark is scalable and can be easily turned into a data-generation pipeline for training.

I think great benchmarks can only be built by crawling real-world data sources, not by manual design.

As models get better, there are less and less ppl that can do tasks that models can't, given a timeframe+budget.

I also believe that real world code mined from real projects that real people developed targeting real objectives will always be better than contractor-procured code

@ChrisPainterYup @hamandcheese what do you mean? just reverse engineer and decompile steam games

@xlr8harder you know it, i know it, everyone knows it

Secret sauce of frontier models.

As models become better they might need less and less task-specific training data so this might change in the future- the value of scalable data-generation pipelines might be much lower when we don't need much data

There's still *a lot* left to train+eval on in the SWE domain, and I'm sure there will be use for humans in the next 2-3 years, but long term I believe that the best approaches will come from scalable, natural data sources and not from humans.

SWE-bench showed the eval+train potential of GitHub issues. ProgramBench showed the potential of terminal programs. We'll find more of these amazing sources and figure out how to harness them.

@xeophon I'm not sure I actually do (unless the answer is that it doesn't.)

Crowd sourcing could do some of this but all of the efforts so far have been quite lackluster.

Oh, just distillation, I guess?

That's why we've always strived to find data in the real world: For SWE-bench, another approach could've been to make our own bugs+fixes in repos we would have written- that would have been so artificial, of such lower quality, and also probably much easier to solve.

@xlr8harder @xeophon you don’t have to

closed model needs to be good at every use case

open model needs to be good at your use case

just do last-mile training on your own tasks and data

@xlr8harder ding ding ding

@teortaxesTex In what way? Surely if the demand is there the market will begin producing the good soon

@cwolferesearch I don’t think that task size is a good proxy for eval quality, esp with long horizon tasks

The same is true for ProgramBench. We used real programs instead of just constructing our own or paying crowd-workers to construct some This lead to a high-quality benchmark with tools like FFmpeg and the PHP compiler- no amount of money could've gotten us that from a contractor

@teortaxesTex time to double down on http://ulam.ai/data

@ChrisPainterYup Is this because of budget constraints, or because METR has different requirements than frontier labs for this kind of data?