Researcher Argues AI Safety Needs Deliberate Work, Not Automatic Convergence

Actual coordination and cooperation among agents have a lot of structure. When working well it is an attractor state - but sadly not the global attractor, as the world shows. We should expect safe ASI to be at least as tricky to make as an ASI-regulating treaty.

Treaties to ban superintelligence are unlikely to be permanent - it is very useful. But we should aim to ban unaligned or unalignable superintelligence. With the right framework one can have nuclear energy without weapon proliferation. And treaties can buy much needed time.

Actually constructing a robust cooperative core with ASI is the most transhumanist thing ever; it gives access to abilities far beyond human. Just assuming it happens naturally is like assuming mixing all supplements, drugs and gene therapies will safely extend life.

That is not a reason not to try either of them.