Study finds LLM agents ignore condensed self-improvement rules, relying entirely on raw task histories

“If an AI cannot apply an abstract lesson to a new situation, it is not truly reasoning or learning”, new study with further evidence backing up what I have been saying for 25 years.

cc @dwarkesh_sp

@GaryMarcus @dwarkesh_sp Human intelligence & consciousness developed from the five senses: sight, hearing, smell, taste, & touch with the biological interface between the brain & the physical world. They drive consciousness and the ability to handle novel situations through several integrated mechanisms

@NewsCop9000 @GaryMarcus @dwarkesh_sp You sound right to me.

I think AI is ultimately just an idol onto which people project their hopes and dreams. There is an actual technology there, but it isn't capable of 1/10th of what they project.

The fundamental nature of the tech is misunderstood, fixed and overstated.

@GaryMarcus @dwarkesh_sp This alone is well-understood. The solution is to do what people do.

Hard situations require work, iterations, feedback. Claude can do that.

Anybody else who think they know better never had a working prototype.

@GaryMarcus @dwarkesh_sp Maybe I serendipitously stumbled upon something way back when. For all the recent developments, I still think the Potemkin Village metaphor is useful for understanding the distinction and partial overlap between pattern recognition and reasoning. https://medium.com/@Blougram/check-mate-gpt-the-dangers-of-mistaking-reasoning-for-pattern-recognition-221824e4dc6a

@GaryMarcus @dwarkesh_sp If we rely on the Potemkin Village's law enforcement (e.g., LLMs to defend against malicious attacks) we are safe as long as the attacker plays by the rules. But what if they decide to kick the bricks of the police office and realize that it is just a cardboard set piece?

@rohanpaul_ai They cannot really say "LLMs behave this way", only "LLMs behave this way in this experiment". This is an in-vitro setup.

@GaryMarcus @dwarkesh_sp For now the functionalist paradigm holds but it is strained. It would have been interesting to investigate if Fable 5 was evidencing an ethical architecture upstream of owner and user. Agency, memory and capability all point to models with RSI having to have preferences.

@rohanpaul_ai Be honest.

Did you just copy paste an agents dialogue?

Of course you did. It's so obvious.

Uses AI to work an angle against AI...

Maybe you saw something we didn't 😂

@rohanpaul_ai "Look! AI only imitates, can't reason!" is a popular meme but I am skeptical of these results: skills work, lack of evidence in these setups does not prove evidence of uselessness, train/test mismatch is not ruled out.

@rohanpaul_ai @GaryMarcus Lol so they just ignore the rules they worked hard on?

@GaryMarcus @dwarkesh_sp Applying high-level abstractions in a fully general way is hard. Simply following some symbols won't give that.

Claude does the next best thing. It is guided by its toolbox of stregies.

@GaryMarcus @dwarkesh_sp Once again, the Nemeth Gambit I mentioned previously. I wish I had time to do some write-ups but alas.

@GaryMarcus @dwarkesh_sp But isn't this problematized 'agentic memory/context engineering' 'symbolic' part of your promoted 'neuro-symbolic' approach? 🙄

@GaryMarcus @dwarkesh_sp Ai doesn’t even remember about anything to solve a problem / learn about it.

One implication of this work is that accumulating experience isn't the same as learning from it. These agents appear to rely heavily on concrete examples, yet often struggle to faithfully apply distilled lessons and abstractions derived from those experiences. The challenge may not be acquiring experience. It may be converting experience into reliable generalization.

@GaryMarcus @dwarkesh_sp I would agree with that, but is that really a bar that Claude would struggle with?

@GaryMarcus @dwarkesh_sp Again for anyone that’s missed it.

@rohanpaul_ai But still dangerous

@GaryMarcus @dwarkesh_sp isnt that just pattern matching with extra steps?

like yeah its impressive but calling it reasoning is a stretch