Elizabeth Barnes, AI safety researcher and founder/CEO of METR, says development is likely on track for AI systems capable of causing human extinction or permanent disempowerment possibly within the next few years
Gary Marcus quoted the post in full agreement.
#153Gary Marcus@GARYMARCUS100% agree with @BethMayBarnes on this point and most (though not quite all*) of her important thread.
*i am much less concerned about extinction risk per se, as discussed in my TLS review of If Anyone Builds It.
Elizabeth Barnes@BethMayBarnes(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower
#153Gary Marcus@GARYMARCUSI 100% agree with @BethMayBarnes on this point and most (though not quite all) of her important thread
Elizabeth Barnes@BethMayBarnes(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower
(2) Things are chaotic and rushed; we aren’t on top of the basics (models regularly violate user intent, labs train on things they meant to avoid, security probably isn’t good enough to prevent adversaries stealing dangerous models) let alone thorny questions of how to control/align superhuman AI
Elizabeth Barnes@BethMayBarnes(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years
(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower
Elizabeth Barnes@BethMayBarnes(3) METR (and other independent orgs, as well as safety/security teams at labs) feel woefully under-resourced compared to the scale and pace of AI development - we’re struggling to build benchmarks fast enough, keep ahead of latest capability developments, read and respond to all the safety-related claims that AI developers are making, run all the evaluations and assessments that companies + governments are asking us to, plus develop the science needed to assess risks from increasingly capable AIs.
@BethMayBarnes What does "quite possibly" mean here? Can you be more precise about how likely you think this is to occur within the next few years?
Elizabeth Barnes@BethMayBarnes(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years
I appreciate Beth stating up front how bad the situation is, but I can't help but feel like her prescription is out of step with her diagnosis. If the industry moved at 1/10 the speed, but it was still trying to render us obsolete, that would still be unacceptably dangerous and democratically illegitimate. Moving toward it at a slower pace does not solve the problem.
We should do the obvious thing and just stop. Obvious does not mean simple or easy, but it is doable. I wrote a whole book on why and how (Obsolete: The AI Industry's Trillion-Dollar Race to Replace You—and How to Stop It. Available soon, info in bio).
Elizabeth Barnes@BethMayBarnesSometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO: