If you mean why I think it’s too late to stop, I think we have simply come too far. Mythos-class models exist. The next models beyond them are already in training. Stopping does not uninvent them. A global ban on using them would be difficult to the point of impossibility. At minimum, the world continues from this level.
We are now close to both AI systems matching human AI researchers and then surpassing them, and to closing the loop on RSI. I think both probably happen this year, and almost certainly by 2027. It is close enough now that a sufficiently capable group, or even a rich enough individual, could plausibly get there. The models themselves would help them do it.
Trying to stop at this point would create immense distortions. Nations, groups, and individuals would defect and continue the work covertly. That is much more dangerous than the world we are in now. The global panopticon and enforcement regime required to police this has never seemed feasible to me, for many reasons, not least because the public would resist it. Some nations would resist it, while frantically attempting to advance as fast as possible.
The models are already capable enough that biological, nanotech, quantum, or other novel jagged breakthroughs feel very close. The further we go, the more feasible it becomes for defectors to pursue those breakthroughs at the level of a small group or even an individual. The reason Bostrom gave up on pause is he thinks the bio/nano revolutions now approaching may themselves be extinction-level events without advanced models helping us navigate them.
We have come too far. The technology is too easy to build, and it helps advance itself. We are already too high up. At this point, attempting to stop would itself be catastrophic.
