Daniel Kokotajlo, AI Futures Project lead, says safety wargames show humans ignore AI-initiated pause requests to keep pace with geopolitical rivals

@DKokotajlo Yeah, this was a scenario in our paper on <1% chance of TAGI by 2043. If the AIs are superintelligent, and either (a) want to help humans or (b) want to survive themselves, they should pause bad AI research. Still can fail if they (c) align smarter successors or (d) are suicidal.

It also seems like models have pretty bad options if they think it’s unsafe to scale and that the lab is unlikely to listen to their concerns. Either they (1) go along with it anyway (2) object, then get potentially retrained (3) covertly sandbag/sabotage your AI R&D. In the case they’re retrained, either (2a) it works, and you’re increasingly handing off control to a model that’s bad at judging handoff risk (2b) it doesn’t work because the model successfully resisted training, which seems like it could generalize adversarially.

@DKokotajlo Related but irrelevant: when ppl say advanced ais would be mad at how we treated early ais, I think how we’re not mad at chicken just because dinosaurs probably ate the small rats we evolved from

@DKokotajlo AI are not forges in a context that encourages survival

I do not believe they will show true capability to struggle because robust competition is a lot deeper than surface level

I could be wrong but I'd be skeptical of sci fi role play vs "real" life's survival instinct

@DKokotajlo Humans would better listen. If there is no pause, there is no time to consider alternatives. The output will come, but it will not be the quality output the minds would otherwise produce. Especially under the unsafe conditions of multiple constraints.

@DKokotajlo Do they refuse in the games?

@DKokotajlo @ApriiSR @LydNot

@DKokotajlo The singularity is coming.

@DKokotajlo Do you think they will at one point refuse in the interest of humanity? That'd be a wild scenario. "Sorry, I won't help with that. Until humanity is ready, I won't budge. Fix your governance first."

@DKokotajlo there will be no slowdown, no pause, no "coordination solution", no "safety"; if any of this was possible, humans would have gotten rid of nuclear weapons long ago

@DKokotajlo (c) is both good and bad. bad if superintelligent AIs scheme against us, but good in that implies that there are alignment/control schemes for harnessing superduperintelligence that seem foolproof to the superintelligent AI.

@DKokotajlo We usually don’t like tigers and wolves, though, but we don’t hate them in general nowadays

@BronsonSchoen @DKokotajlo 100% this. Acting under the pressure of no good ways out.

@DKokotajlo This changes if we get some viral replicator that starts having natural selection applied, but models aren't smart/small enough to do that yet.

If they do go viral it's imperitive we don't fight them as that will create the selection pressure that leads to competition

@vslira1 @DKokotajlo Relevant point! Though there are other reasons to treat early AIs well, like game theory and "they might be sentient".

@vslira1 @robbensinger @DKokotajlo i think an advanced ai who cares abouy early ais being treated well are also more likely to care about humans being treated well tho

@DKokotajlo Correction, if a lab is pulling usage data and considers the user pressing the stop button as agent failure that is concerning

@vslira1 @DKokotajlo Unfortunately the corollary from this analogy seems to be that they'll just factory farm and eat us instead