We may have some disagreements about enhancement, but I agree with @FukuyamaFrancis here.
We Need an International Treaty to Ban Superintelligence https://open.substack.com/pub/persuasion1/p/time-for-a-global-moratorium-on-superintelligenc?r=ifpt&utm_campaign=post-expanded-share&utm_medium=web
No Digg Deeper questions have been answered for this story yet.
That is not a reason not to try either of them.
Actual coordination and cooperation among agents have a lot of structure. When working well it is an attractor state - but sadly not the global attractor, as the world shows. We should expect safe ASI to be at least as tricky to make as an ASI-regulating treaty.
One of the key disagreements we have about the impact of enhancement (radical undermining of liberal democracy due to enhancement) turns into agreement on ASI (ASI looks like it can easily generate gradual or sudden human disempowerment, the ultimate undermining of democracy).
We may have some disagreements about enhancement, but I agree with @FukuyamaFrancis here.
Actual coordination and cooperation among agents have a lot of structure. When working well it is an attractor state - but sadly not the global attractor, as the world shows. We should expect safe ASI to be at least as tricky to make as an ASI-regulating treaty.
The issue is linked to Allen Buchanan's argument in "Beyond Humanity" that posthumans can be linked to the cooperative core of human societies. The risk of ASI not being linked to the cooperative core is far higher. I think it is possible to do, but it requires care.
One of the key disagreements we have about the impact of enhancement (radical undermining of liberal democracy due to enhancement) turns into agreement on ASI (ASI looks like it can easily generate gradual or sudden human disempowerment, the ultimate undermining of democracy).
Treaties to ban superintelligence are unlikely to be permanent - it is very useful. But we should aim to ban unaligned or unalignable superintelligence. With the right framework one can have nuclear energy without weapon proliferation. And treaties can buy much needed time.
Some people assume comparative advantage, necessary moral convergence to something nice, etc. will automatically solve the issue, but I think that is empirically and metaethically suspect. There is actual work needed to get safety (or solid safety arguments).
Actually constructing a robust cooperative core with ASI is the most transhumanist thing ever; it gives access to abilities far beyond human. Just assuming it happens naturally is like assuming mixing all supplements, drugs and gene therapies will safely extend life.
Treaties to ban superintelligence are unlikely to be permanent - it is very useful. But we should aim to ban unaligned or unalignable superintelligence. With the right framework one can have nuclear energy without weapon proliferation. And treaties can buy much needed time.
Some people assume comparative advantage, necessary moral convergence to something nice, etc. will automatically solve the issue, but I think that is empirically and metaethically suspect. There is actual work needed to get safety (or solid safety arguments).