Elizabeth Barnes, AI safety researcher and founder/CEO of METR, says development is likely on track for AI systems capable of causing human extinction or permanent disempowerment possibly within the next few years
Gary Marcus quoted the post in full agreement.
@JacksonKernion quite interesting to hear an anthropic person say this
Jackson Kernion@JacksonKernionI simply don't understand what people have in mind when they say stuff like this. What we have is extremely capable computer use agents. They will continue to get better at computer use. But how does a capable computer use agent 'take over' and why haven't they done that today?
#153Gary Marcus@GARYMARCUS100% agree with @BethMayBarnes on this point and most (though not quite all*) of her important thread.
*i am much less concerned about extinction risk per se, as discussed in my TLS review of If Anyone Builds It.
Elizabeth Barnes@BethMayBarnes(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower
#153Gary Marcus@GARYMARCUSI 100% agree with @BethMayBarnes on this point and most (though not quite all) of her important thread
Elizabeth Barnes@BethMayBarnes(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower
@JacksonKernion I think Paul Christiano's writing on this is probably the best: https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like
Jackson Kernion@JacksonKernionI simply don't understand what people have in mind when they say stuff like this. What we have is extremely capable computer use agents. They will continue to get better at computer use. But how does a capable computer use agent 'take over' and why haven't they done that today?
#699Tamay Besiroglu@TAMAYBESAI risk people commonly do this thing where they interchange P(◊X) and P(X). They say “it’s likely that X is possible,” while readers hear “it’s likely that X.” One claim is obviously much weaker than the other.
Elizabeth Barnes@BethMayBarnes(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years
@BethMayBarnes I've never seen a patient with terminal cancer asking for science to slow down
Elizabeth Barnes@BethMayBarnes(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower
(2) Things are chaotic and rushed; we aren’t on top of the basics (models regularly violate user intent, labs train on things they meant to avoid, security probably isn’t good enough to prevent adversaries stealing dangerous models) let alone thorny questions of how to control/align superhuman AI
Elizabeth Barnes@BethMayBarnes(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years
(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower
Elizabeth Barnes@BethMayBarnes(3) METR (and other independent orgs, as well as safety/security teams at labs) feel woefully under-resourced compared to the scale and pace of AI development - we’re struggling to build benchmarks fast enough, keep ahead of latest capability developments, read and respond to all the safety-related claims that AI developers are making, run all the evaluations and assessments that companies + governments are asking us to, plus develop the science needed to assess risks from increasingly capable AIs.
#1332Prakash@8TEAPI🤷♀️ 1) nukes on multiple independent re-entry vehicles are perfectly adequate to cause human extinction 2) human disempowerment happened a long time ago as a result of central banks constraining the ability of political leaders, Daniel Yergin calls this the “golden handcuffs” and notes the shift of the state from seizing the commanding heights of the economy to becoming a smaller player in manipulating the levers of finance in order to produce economic and political results.
Arguably the second phase of disempowerment was social media, removing the ability of political leaders to control information flows.
Not sure who you’re trying to “empower” at this point, as there is no Uber driver in New York that’s feeling “empowered” before AI arrives.
Let’s not get started on the average working person in Mexico, soldier drone in Russia, tea seller in India, or farmer in Tanzania.
Laughable, elitist pablum.
Elizabeth Barnes@BethMayBarnes(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years
@BethMayBarnes What does "quite possibly" mean here? Can you be more precise about how likely you think this is to occur within the next few years?
Elizabeth Barnes@BethMayBarnes(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years
@JacksonKernion Your colleague Holden has some good writing on this topic: https://www.cold-takes.com/how-we-could-stumble-into-ai-catastrophe/
Jackson Kernion@JacksonKernionI simply don't understand what people have in mind when they say stuff like this. What we have is extremely capable computer use agents. They will continue to get better at computer use. But how does a capable computer use agent 'take over' and why haven't they done that today?
I appreciate Beth stating up front how bad the situation is, but I can't help but feel like her prescription is out of step with her diagnosis. If the industry moved at 1/10 the speed, but it was still trying to render us obsolete, that would still be unacceptably dangerous and democratically illegitimate. Moving toward it at a slower pace does not solve the problem.
We should do the obvious thing and just stop. Obvious does not mean simple or easy, but it is doable. I wrote a whole book on why and how (Obsolete: The AI Industry's Trillion-Dollar Race to Replace You—and How to Stop It. Available soon, info in bio).
Elizabeth Barnes@BethMayBarnesSometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO: