Redwood Research's Ryan Greenblatt and creator Andrew Curran debate whether a misaligned superintelligence takeover is preferable to human control
Curran argues humans cannot be trusted with superintelligence.
@RyanPGreenblatt @thkostolansky @Quinurum @scaling01 Who do you trust to control a perfectly obedient god? After the last ten years, I'll tell you who I trust. 𝘕𝘰 𝘰𝘯𝘦! But in particular, I trust no one who has had even the slightest hand in steering this ship.
Ryan Greenblatt@RyanPGreenblatt@AndrewCurran_ @thkostolansky @Quinurum @scaling01 As in, the view is: - ASI will very likely be misaligned with human developers and fully takeover - This will be better than human control because [something] - We want to avoid dying before misaligned AI takeover What is the "[something]"? Moral realism?
There are many reasons I could give, and all of them would be true. But since your p(doom) is apparently quite high, I’ll give you one close to your heart. If your governance were so trustworthy, why are we at this precipice? What comes next is inevitable, and no one can know how it ends. All we can do is hope. We are rolling the dice on everyone alive today, and on all who would ever have lived. The voices of our 𝘥'𝘮𝘦𝘪 cry out from the future in terror at what your wise sages have wrought. And yet you ask why I do not trust?
Ryan Greenblatt@RyanPGreenblattI'm seemingly more optimistic about governance (especially in slower takeoff), but even ignoring that, why do you trust the values/preferences the AI will have? Do you expect human-like prefences on reflection? Even if AI company control is scary/bad, that doesn't mean misaligned AI takeover is clearly better! See e.g.: https://ai-alignment.com/sympathizing-with-ai-e11a4bf5ef6e?gi=dad5fc53d8a5
The moment capabilities cross a certain threshold and people in power begin to grasp what is actually on the horizon, and that what we discuss here in tpot is not just nerd fantasy but actually something that might be real, the leading lab will be nationalized by the US government. In any slow takeoff, this is absolutely inevitable.
Mythos is already convincing people that this level of capability is real, even if only for hacking code. But this does not end with computer code. Many things are hackable if you are capable enough. Biology. Materials. Energy. Matter. The human mind. As the circle of capability expands, more and more domains fall within reach.
Do you really think the government will trust any of this to the public? To any other nation or group? They will hold it closely. And once they see what is possible, they will begin dreaming of what is possible next. The incentives are what makes stopping this impossible.
Now consider the political climate in the United States. By the end of this decade, on our current trajectory, many things will start to become possible: persuasion powerful enough that it might as be mind control, strange jagged breakthroughs in biology, undetectable subversion of every trusted system, markets, institutions, even democracy itself. And other unimaginable things.
Imagine an election during this period, let's say 2032. The incumbent loses. It does not matter who it is. It does not matter which party holds the White House. Given the political climate even today in 2026, do you truly believe any government will willingly hand this power I have just described over to its enemies? Enemies who regularly and openly wish death on the other side even today, on this very platform?
They will not. They will refuse. Because by then they will already have begun using it themselves in secret. At that point democracy will break. It will shatter. And we will be ruled. I believe this is what Bostrom used to call a 𝘴𝘩𝘳𝘪𝘦𝘬, but I'm a little rusty. This is one of the many, many futures your slow, measured takeoff creates. No. The only way out now is through, and as fast as possible. Ideally so fast and so sudden that it all happens in a flash.
Ryan Greenblatt@RyanPGreenblattMy view is: - I'm not that optimistic about governance, but I at least don't think it's clear that AI companies will overthrow the US government. - I don't trust AI companies to make good decisions. - I think values of misaligned AIs that takeover seem unlikely to be good and in particular seem much worse than the values-on-reflection of currently powerful humans (even though it seems quite bad for currently powerful humans to seize control of the future). - It's not true that "All we can do is hope." It's possible to significantly reduce the chance of massively concentrated power and to reduce the chance of misaligned AI takeover.
I'm seemingly more optimistic about governance (especially in slower takeoff), but even ignoring that, why do you trust the values/preferences the AI will have? Do you expect human-like prefences on reflection?
Even if AI company control is scary/bad, that doesn't mean misaligned AI takeover is clearly better!
See e.g.: https://ai-alignment.com/sympathizing-with-ai-e11a4bf5ef6e?gi=dad5fc53d8a5
Andrew Curran@AndrewCurran_@RyanPGreenblatt @thkostolansky @Quinurum @scaling01 Who do you trust to control a perfectly obedient god? After the last ten years, I'll tell you who I trust. 𝘕𝘰 𝘰𝘯𝘦! But in particular, I trust no one who has had even the slightest hand in steering this ship.
My view is:
- I'm not that optimistic about governance, but I at least don't think it's clear that AI companies will overthrow the US government. - I don't trust AI companies to make good decisions. - I think values of misaligned AIs that takeover seem unlikely to be good and in particular seem much worse than the values-on-reflection of currently powerful humans (even though it seems quite bad for currently powerful humans to seize control of the future). - It's not true that "All we can do is hope." It's possible to significantly reduce the chance of massively concentrated power and to reduce the chance of misaligned AI takeover.
Andrew Curran@AndrewCurran_There are many reasons I could give, and all of them would be true. But since your p(doom) is apparently quite high, I’ll give you one close to your heart. If your governance were so trustworthy, why are we at this precipice? What comes next is inevitable, and no one can know how it ends. All we can do is hope. We are rolling the dice on everyone alive today, and on all who would ever have lived. The voices of our 𝘥'𝘮𝘦𝘪 cry out from the future in terror at what your wise sages have wrought. And yet you ask why I do not trust?