Alex Dimakis, UC Berkeley professor, says AI coding agents scale logarithmically with compute while human experts scale super-linearly

@lihanc02 Thanks for sharing! So far, humans still seem much better at test-time strategy and context management. Hard to imagine what happens once AI can scale at test time like humans.

Aside from what the blog says, my main takeaway when Qiuyang showed me the results is: human brains are actually so heavily optimized for context management that almost no LLMs could compare...

This disputes infinite token scaling and actually completes my personal view on AI

@MangQiuyang @lihanc02 Any thoughts on better (agent/LLM) inference time scaling?

@lihanc02 Yeah to me it seems more like humans have short context windows (but very information dense) and the brain constantly works to put the right things in context every second.

@AlexGDimakis I think a simpler explanation would be: agents find some idea and spend all the iterations generating small perturbations that marginally increase the score while a human would dedicate a part of his time to finding genuinely new ideas.

@MangQiuyang @AlexGDimakis yes! i have been trying to put together useful bmk data for context management but it has been quite tough. i think the first step is always to be able to measure the impact.

I have the same thought: bad context management is essentially a bad test-time strategy. Within a single context window, we have already seen some models exhibit superlinear scaling laws. So this may become a research question that sits at the intersection of model architecture for longer context window and the intra-window context management.

@lihanc02 We're so good at coming up with efficient representations of data to think about so our thinking is a lot more efficient than LLMs

@JayooHwang Yep I think humans and AI simply have different thinking patterns

@AlexGDimakis Fascinating find!

@AlexGDimakis arguably there is another model of this; the agent scales super-linearly as it gets useful context, but then there is a degradation term of context rot / forgetting stuff. that could lead to the same results. i.e. if the context window were really 10M it would look superlinear.

@xwangsd @lihanc02 I think @yoonholeee’s MetaHarness paper is highly relevant here: https://yoonholee.com/meta-harness/

@AlexGDimakis interesting framing so brute force approach scales predictably but peaks, while structured reasoning keeps growing

@PandaAshwinee @MangQiuyang I think this plot shows that after 4 hours context management becomes the bottleneck and Claude code becomes random sampling of solutions.

@AlexGDimakis This matches what I feel when using coding agents. They are great at fast starts, but long work still needs taste, memory, and many small human decisions

@MrCatid yeah I think maybe one day we will see swe-> context engineers who just design context management mechanisms for specific tasks

@AlexGDimakis Btw the comparison is imo a bit dishonest. Gpt 5.5 + a good harness is already better than all human contestants on AGC. I'm pretty sure a good harness for AHC would lead to similar results.

@AlexGDimakis the plateau part matches what ive been seeing in practice

interesting that random sampling still scales linear tho

@lihanc02 metamemories on metamemories in metamemories...

@xwangsd @MangQiuyang I think maybe the end goal is to get a better context management similar to human or sth