20h ago
Eli Lifland, Samotsvety Forecasting co-lead, argues Anthropic's automated AI research will bottleneck on compute once systems surpass human judgment
Daniel Kokotajlo warned recursive self-improvement presents extreme dangers.
Daniel Kokotajlo warned recursive self-improvement presents extreme dangers.
I believe the majority still doesn't understand the momentous threshold humanity is facing. Anthropic itself states quite clearly that even if development ceased entirely, if all development were frozen, they would still witness massive societal changes: "Even if model capabilities were frozen at today’s level, we would expect major changes to occur in the world. (...) And we are still early in the diffusion of today’s models into the wider economy, where a 100-person company can increasingly do the work of a 1,000-person one, because each employee will sit atop a pyramid of agents." But there's no question of stagnation. Anthropic itself still maintains that development has exceeded its own internal assumptions. Take that statement seriously for a second and consider it. Although Anthropic models internally and assumes exponential development, even this trajectory lags behind actual development, which is even faster. "It's happening faster than we thought, and the implications deserve greater attention." and "The rate at which AI models improve is accelerating. The length of tasks that they can reliably complete on their own has been doubling roughly every four months, up from an earlier trend of doubling every seven months. In March 2024, Claude Opus 3 could complete software tasks that take humans about four minutes to complete. A year later, Claude Sonnet 3.7 managed tasks that took about an hour and a half. A year after that, Claude Opus 4.6 managed 12-hour tasks.1 If this trend holds, tasks that take a skilled person days could come into range this year. So again: there can be no question of standing still. The models are not only getting better, they can also work autonomously for longer. Certainly numerous breakthroughs are still needed, context window is still a problem. But the most likely direction is that the models themselves will find the solutions to the underlying problems. This opens up unforeseen possibilities, and Demis Hassabi's statement that the golden age of science is not a dream, not a utopia, but a purposeful reality, is now confirmed. And finally, it's not just Anthropic, but also OpenAI, that sees this development, considers it feasible, and is moving forward. Most people don't know what's coming. But one thing is certain: it's coming even faster than expected. And it will be even bigger. Myth was just the beginning.
