Research co-authored by Alex G. Dimakis finds human experts scale super-linearly while AI coding agents scale linearly with test-time compute

@AlexGDimakis this is cool

It’s an easy proof: assume the performance in a task is a Gaussian zero mean with variance 1. Sample it k times. Random sampling is taking the max performance from k tries. Elo can be computed by the probability of k1 trials beating k2, ie the max of k1 Gaussians to be bigger than the max of k2 gaussians. Elo(k) = constant + log(k)

@AlexGDimakis arguably there is another model of this; the agent scales super-linearly as it gets useful context, but then there is a degradation term of context rot / forgetting stuff. that could lead to the same results. i.e. if the context window were really 10M it would look superlinear.

@AlexGDimakis Btw the comparison is imo a bit dishonest. Gpt 5.5 + a good harness is already better than all human contestants on AGC. I'm pretty sure a good harness for AHC would lead to similar results.

@AlexGDimakis the plateau part matches what ive been seeing in practice

interesting that random sampling still scales linear tho

very interesting analysis paradigm!

> unless we solve continual learning we will not be able to outperform humans in tasks that take many days

I’m not sure this follows. In general yes, if humans scale superlinearly vs agents they win eventually, but for any fixed horizon it may be that agents have a sufficiently good scaling coefficient to outperform humans when t < horizon; the human horizons-by-task probably drop off such that even constant-scale agents will be good-enough replacements for most things

@AlexGDimakis I think a simpler explanation would be: agents find some idea and spend all the iterations generating small perturbations that marginally increase the score while a human would dedicate a part of his time to finding genuinely new ideas.

@AlexGDimakis This matches what I feel when using coding agents. They are great at fast starts, but long work still needs taste, memory, and many small human decisions

@AlexGDimakis Agents seem to be submodular, humans supermodular.

@AlexGDimakis interesting framing so brute force approach scales predictably but peaks, while structured reasoning keeps growing

@MangQiuyang @AlexGDimakis I think that on the long run, the human curve is more of a staircase with exponentially longer plateaus: it requires a really novel idea to make significant improvements, and these ideas are exponentially harder to find. Also, if the agent can get high enough fast

@AlexGDimakis What tools did the agents have access to? Could they build their own repository of knowledge through the run?

@obsolete26 It’s Claude code so I can do whatever it wants. But it doesn’t.

@MangQiuyang @AlexGDimakis yes! i have been trying to put together useful bmk data for context management but it has been quite tough. i think the first step is always to be able to measure the impact.

I have the same thought: bad context management is essentially a bad test-time strategy. Within a single context window, we have already seen some models exhibit superlinear scaling laws. So this may become a research question that sits at the intersection of model architecture for longer context window and the intra-window context management.

@AlexGDimakis So full bash, file system and web access?

@AlexGDimakis Is it really memoryless?

I don’t know whether GPT has an optimized harness for AGC or general CP problems. But I’m wondering whether agents’ test-time scaling ability will eventually depend on task-specific harnesses.

For humans, AHC and AGC participants are largely from the same community, and many of them have also trained extensively on traditional CP problems. Btw, human are not solving the problems under the full contest time window.

@AlexGDimakis By "continual learning", do you mean "adjustment of weights in realtime"?

@_Yann77 @AlexGDimakis Our work about CP harness may be related https://arxiv.org/abs/2605.15177. I agree a good harness can change the final Elo, but I don't know if they can also change curve and complexity?