The call drew support from MIRI and academic researchers.
We Need an International Treaty to Ban Superintelligence https://open.substack.com/pub/persuasion1/p/time-for-a-global-moratorium-on-superintelligenc?r=ifpt&utm_campaign=post-expanded-share&utm_medium=web
Many users backed the global treaty to ban superintelligence for safety reasons while others rejected it as crippling AI progress and insulted its proponents.
No Digg Deeper questions have been answered for this story yet.
The problem we face with AI today is not a technical problem, it is a political problem.
Of who gets to decide. What level of risk the public is exposed to, what future we build towards or avert.
It's so heartening and important to see this conversation starting to happen in the world outside the very insular tech futurist bubble. We need to have these conversations, everywhere, and this piece by @andreamiotti hosted by Francis Fukuyama I hope is a very useful step in that direction!
We Need an International Treaty to Ban Superintelligence https://open.substack.com/pub/persuasion1/p/time-for-a-global-moratorium-on-superintelligenc?r=ifpt&utm_campaign=post-expanded-share&utm_medium=web
We may have some disagreements about enhancement, but I agree with @FukuyamaFrancis here.
We Need an International Treaty to Ban Superintelligence https://open.substack.com/pub/persuasion1/p/time-for-a-global-moratorium-on-superintelligenc?r=ifpt&utm_campaign=post-expanded-share&utm_medium=web
@FukuyamaFrancis No, we don't.
That's like saying in 1920 "we need an international treaty to ban jet engines"
We Need an International Treaty to Ban Superintelligence https://open.substack.com/pub/persuasion1/p/time-for-a-global-moratorium-on-superintelligenc?r=ifpt&utm_campaign=post-expanded-share&utm_medium=web
Fuck off.
We Need an International Treaty to Ban Superintelligence https://open.substack.com/pub/persuasion1/p/time-for-a-global-moratorium-on-superintelligenc?r=ifpt&utm_campaign=post-expanded-share&utm_medium=web
the sequel to The End of History is apparently Actually, Let’s Not.
We Need an International Treaty to Ban Superintelligence https://open.substack.com/pub/persuasion1/p/time-for-a-global-moratorium-on-superintelligenc?r=ifpt&utm_campaign=post-expanded-share&utm_medium=web
@ylecun @FukuyamaFrancis Im not sure jet engines and superintelligence deserve equivalence
No, the chain of today’s top AI might be narrow, but there is a vast space of possible ways to use computers to make AI. So you’d have to give a world govt strong power over all computers to block AI progress. What else would it do with such power?
@ylecun @FukuyamaFrancis What's an example of an observation that would meaningfully shift you on this issue? Last I checked, you think LLM-ish AIs won't scale to superintelligence, but you think it would be very dangerous if they did. Is there a concrete capability you're confident LLMs won't exhibit?
One of the key disagreements we have about the impact of enhancement (radical undermining of liberal democracy due to enhancement) turns into agreement on ASI (ASI looks like it can easily generate gradual or sudden human disempowerment, the ultimate undermining of democracy).
We may have some disagreements about enhancement, but I agree with @FukuyamaFrancis here.
The issue is linked to Allen Buchanan's argument in "Beyond Humanity" that posthumans can be linked to the cooperative core of human societies. The risk of ASI not being linked to the cooperative core is far higher. I think it is possible to do, but it requires care.
One of the key disagreements we have about the impact of enhancement (radical undermining of liberal democracy due to enhancement) turns into agreement on ASI (ASI looks like it can easily generate gradual or sudden human disempowerment, the ultimate undermining of democracy).
@ylecun @FukuyamaFrancis I support yann here
Superintelligence can create more jobs that we can’t even imagine now
Treaties to ban superintelligence are unlikely to be permanent - it is very useful. But we should aim to ban unaligned or unalignable superintelligence. With the right framework one can have nuclear energy without weapon proliferation. And treaties can buy much needed time.
Some people assume comparative advantage, necessary moral convergence to something nice, etc. will automatically solve the issue, but I think that is empirically and metaethically suspect. There is actual work needed to get safety (or solid safety arguments).
@anderssandberg @FukuyamaFrancis You can’t ban what you can’t define. We could try to ban AG (anti gravity) but since we don’t have a mechanism what do we ban?
@FukuyamaFrancis Thank you.
We also need international agreed standards for company valuations.
A company’s valuation should not be allowed to float endlessly on hype, narrative, or speculative future dominance. As a baseline discipline, no company valuation should exceed 100% of its average annual revenue over the previous ten years unless it can justify the premium through audited profitability, assets, defensible cash flow, or proven market control.
@FukuyamaFrancis Why not also ban unicorns while we are at it? - nothing breaks economist/political scientists like AI.
Actually constructing a robust cooperative core with ASI is the most transhumanist thing ever; it gives access to abilities far beyond human. Just assuming it happens naturally is like assuming mixing all supplements, drugs and gene therapies will safely extend life.
Treaties to ban superintelligence are unlikely to be permanent - it is very useful. But we should aim to ban unaligned or unalignable superintelligence. With the right framework one can have nuclear energy without weapon proliferation. And treaties can buy much needed time.
Again, in order for this to happen, we need to reinvest in our institutions, including, and most importantly, the UN. I've heard every criticism of the UN under the sun, and I share many of them. Unlike many, I was excited to see if the "move fast and break things" Trump administration could do things differently, but none of their alternatives to the UN have worked out.
