tl;dr: we aren’t close to RSI, regardless of the hints IPO-bound Anthropic tried to drop last week.

The hard part is not designing agents that iterate. The hard part is knowing when to stop. Agents without human oversight that debug in a loop will find bugs in their objective, not the world. They will optimize for the proxy, not the goal. Human-in-the-loop is not a limitation, it's supervision.

@GaryMarcus I agree, because just because Claude is now contributing to his own development doesn't mean he can successfully take on this task autonomously without human insight

@GaryMarcus 'IPO-bound Anthropic' assumes the data is a marketing move. But 80% merged-code comes from Dario's own all-hands. 3x to 52x optimization in 12 months against a 4x human ceiling. That's a specific disclosure for a team that's just pitching.

@rohanpaul_ai Limited model calls + hidden tests is the right pain. Most agent evals let them brute force until something passes, then call it autonomy. Once budget matters, they look like junior scripts with a very expensive retry loop.

@rohanpaul_ai This is a useful reality check. Execution ability is improving faster than true system design autonomy.

@ArnoldKuepo @GaryMarcus I think there may be another problem. Recursion presumably cuts both ways. If it designs and runs the wrong experiment, or it gets poisoned, you could get recursive deterioration. You're one outbreak of workslop away from turning Claude into Tay.

@rohanpaul_ai Autonomy means more than just doing tasks well

@GaryMarcus This measures the ability of agents to write a scaffold for a smaller model to achieve better results on a benchmark compared to the best human created scaffold. The ai is under time constraints (unlike the humans). There is no test of the agents machine learning abilities at all

@GaryMarcus This raises another question: how hard can you trust an LLM to improve an LLM?

AISI showed ~250 AI-gen documents can poison a 13B parameter model. LLMs by definition hallucinate eventually. You're using a hallucinating machine to stop a hallucinating machine from hallucinating.

@rohanpaul_ai The 'boring reliability' it's missing isn't a capability gap, it's a boundary gap. Restraint, budget discipline, knowing when to stop aren't things a bigger model hands you. They're constraints enforced from outside the agent and proven to have held. That's executor vs engineer.

@rohanpaul_ai Betting this reliability gap closes faster than expected. Raw capacity is there in frontier models. Scaffolding just needs to catch up.

@rohanpaul_ai not without well embedded secure policies in their control plane. I'm writing more about it here

@rohanpaul_ai The gap between solving a task and designing a reliable solution for that task is still significant.

@GaryMarcus Market's still just chasing narratives rather than actual substance. Typical.

"Powerful executors with flashes of design judgment" is the most precise description I've read. The missing piece is metacognition — knowing when your current approach is fundamentally broken, not just locally improvable. That's what separates an engineer from a smart autocomplete.

@GaryMarcus You're right to be skeptical. But the RSI blocker isn't capability. It's liability and regulatory approval. In robotics, those move way slower than AI breakthroughs. That's the real timeline.

@rohanpaul_ai Asking a mouse to design a better mousetrap. It'll be 'optimized' for the mouse, but are we improving engineering or just automating failure?

@rohanpaul_ai "Powerful executors with flashes of design judgment, still missing the boring reliability that makes engineering real" is the most accurate description of current AI agents I've read