METR economist Tom Cunningham argues AI agents face faster diminishing marginal returns on expenditure than human labor

@testingham how does this account for human-AI teaming? the paradigmatic example is a task which would have taken a human 6 months, but the agents can 80% do it in a day, then the human takes 2 weeks to finish it off collaboratively with the agent

IMO a lot of the frontier is currently this

@testingham where would the human + agent line be in your opinion?

@UriGil3 @testingham if you look at the agent line it still grows even if slower, you can still throw more money and compute into it. point it into auto research à la Karpathy and make it try to improve that curve, make AI eventually bootstrap AGI, probably the point of hiring him.

does it work? idk

Corollary: for most quality levels, either an agent can do it much cheaper than a human, or it can't do it at all.

This has a nice implication: if expenditure is allocated optimally then the value/cost ratio is much *higher* for agents than for humans. So we might see a small expenditure-share on agentic labor, even when they're producing a lot of value.

A test for this: if you doubled your token use, how much would you increase the value you get from AI? This gets elasticity.

My guess would be it's much less than double. (and if you don't usually hit your token limits then implied marginal value is zero).

Inference-time scaling rotates the red curve upwards, increasing the elasticity.

But there's also the countervailing force of distillation: once an agent solves a problem once, it then becomes cheap to do it again.

@testingham I think most serious people expects that eventually the agent's value surpass humans at all points on this graph. the critical question is how fast we progress to that point.

@testingham Actually in IT its actually a convex increasing function the green one if you decide to only pick expensive and good people

@testingham Based on what data? How much investment has the industry in question made into connectors for the agents to the industry data, etc?

Imo connectors are still early-stage

@testingham that's why I use a heterogenous mix of humans and AI agents and distribute tasks across them

@testingham At a firm level, the returns for marginal human labor diminish much more quickly than they do in the broader labor market, right?

So narrowly, firms may see this tradeoff differently, and labor friction prevents the current market from acting like this graph would imply?

@testingham agent cost per task has been halving every 2 months so have to prepare for when they do become cheaper

@testingham you mean human is the wizard and agent the knight?

@testingham @METR_Evals I agree. However, if you extrapolate improvements in model/agent, does your conclusion change?

@testingham @jasonfurman The mechanical reason behind your chart is in this chart I made for a 3 part series on AI.

@testingham And you can't have sex with agents, so there's that, also.

@testingham What about the joined line of one human using AI?

@testingham the curve flattens at high quality because agents stack failure modes the same way each time. humans try 50% differently. agents try 50% harder. that asymmetry is what the chart compresses.

@testingham AI labs are eating your lunch