@Dan_Jeffries1 Did some exponential-pilled bros finally realize that real-world processes have irreducible time constants and that you can't run the real world faster than real time?
This is the best and most balanced report I've read by Anthropic, free of many of the super sci-fi, everything-is-exponential language of some other reports I've read by this amazing team.
But one line is dead wrong. This one about recursive self-improvement:
"[If] AI systems themselves become capable of full recursive self-improvement, and begin building their successors...In this world, the pace of progress in AI development becomes determined entirely by the availability of compute (or the speed of discovering various efficiencies in algorithmic training or inference) for AI systems."
Compute is absolutely NOT the only limiting factor in recursive self-improvement and not even the most important one. They are two more:
1) Time 2) Multiplicity
Time is how long it takes to get an answer.
Multiplicity is when there is no right or wrong answers but only shades of gray with right(ish) answers and wrong(ish).
They even point to one of them (time) just a few paragraphs later:
"More intelligence can’t learn what a drug does over decades of use, can’t hold elections sooner than a constitution dictates, and can’t turn a stranger into an old friend in a weekend. For most people, the felt pace of this future will still be set by the bottlenecks, even if the laboratory upstream runs at the speed of compute. That collision, where recursive intelligence building itself ever faster meets the world of humans, relationships, and governance, is another part of this future we can’t predict."
But let me make it even more clear:
AI got good at code and games because they have great feedback loops and tight timelines. If the code works or does not, you know pretty quickly.
It good at driving for the same reasons. Don't die or drive off the road or hit someone are achievable (though difficult) goals with clear, fast feedback.
You cannot answer the question "is this a good article?" or "do I write well?" because that is multiplicity, shades of gray that are hard to judge.
Humans judge this by self-awareness and feedback from others. AI might be able to approximate the second but only if it develops more of the first (harder).
"Will my wife like this surprise present?" Hard to get good at that even if you're a master. Took me many years of trying and judging her responses. :)
Time is also a massive factor. The question of "did I make money in business?" can't be answered in a short time line. There is no way to know the answer faster, and short term success doesn't predict long term.
"Will this drug cause bad side effects twenty years from now?" That can only be answered in twenty years. No amount of compute changes that.
"Will this building fall down faster than this one if I build it a different way?" You can run basic physics and math rules to help you heuristically figure it out, but only time gives you the true answer.
These two constraints, time and multiplicity, are the death knell of any Doomer/Less Wrong fantasies about fast takeoff and instant super genius AI. You can have all the compute in the universe and you still can't compress twenty years of drug side effects into twenty minutes.
You can have a 500 trillion parameters and you still can't definitively answer "is this beautiful?" because beauty is not a optimization target with a clean gradient.
The recursive self-improvement loop doesn't hit a wall because of compute.
It hits a wall because of reality.
Reality is slow, messy, ambiguous, and full of questions that only time and lived experience can answer.
Compute is the bottleneck that engineers see because it's the one they can measure.
Time and multiplicity are the bottlenecks that the real world imposes and no amount of silicon can brute force past them.
That's why even nature only "solved" good/bad by brute force: evolution. Does this agent/human/creature survive and reproduce? That's good. Otherwise not good.
Companies follow the same rule. Did this survive and make money over time? Good. Otherwise bad.
Imperfect, lossy, dumb, blind, slow.
AI is changing the world already.
It will get better and better.
But the road to better is long and winding, not a vertical line to godhood.
And that should make you more hopeful, not less.
