METR founder Elizabeth Barnes warns that rushed development and narrow evaluations leave labs unable to control catastrophic AI risks

VIEWS235.9KBOOKMARKS89REPLIES37

(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years

Elizabeth Barnes@BethMayBarnes

Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO:

38d235.9K48389

LIKES562

roon@tszzl

@JacksonKernion quite interesting to hear an anthropic person say this

Jackson Kernion@JacksonKernion

I simply don't understand what people have in mind when they say stuff like this.

What we have is extremely capable computer use agents. They will continue to get better at computer use. But how does a capable computer use agent 'take over' and why haven't they done that today?

38d18.3K56225

RETWEETS157

Elizabeth Barnes@BethMayBarnes

Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO:

38d222.7K1K380

Miles Brundage@Miles_Brundage

Yeah, I don't think I've ever met someone who has directly worked for an extended period on safety or security at an AI company who thinks things are fine readiness wise or incentive wise etc.

Elizabeth Barnes@BethMayBarnes

Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO:

38d27.1K27447

Elizabeth Barnes@BethMayBarnes

(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower

Elizabeth Barnes@BethMayBarnes

(3) METR (and other independent orgs, as well as safety/security teams at labs) feel woefully under-resourced compared to the scale and pace of AI development - we’re struggling to build benchmarks fast enough, keep ahead of latest capability developments, read and respond to all the safety-related claims that AI developers are making, run all the evaluations and assessments that companies + governments are asking us to, plus develop the science needed to assess risks from increasingly capable AIs.

38d33.2K40424

Elizabeth Barnes@BethMayBarnes

(3) METR (and other independent orgs, as well as safety/security teams at labs) feel woefully under-resourced compared to the scale and pace of AI development - we’re struggling to build benchmarks fast enough, keep ahead of latest capability developments, read and respond to all the safety-related claims that AI developers are making, run all the evaluations and assessments that companies + governments are asking us to, plus develop the science needed to assess risks from increasingly capable AIs.

38d24.8K34122

Elizabeth Barnes@BethMayBarnes

(2) Things are chaotic and rushed; we aren’t on top of the basics (models regularly violate user intent, labs train on things they meant to avoid, security probably isn’t good enough to prevent adversaries stealing dangerous models) let alone thorny questions of how to control/align superhuman AI

Elizabeth Barnes@BethMayBarnes

(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years

38d25.4K34617

Elizabeth Barnes@BethMayBarnes

Also: many things are out of scope! Firstly, we only consider “AI takeover” / “loss of control” risks: we don’t consider risks from human misuse (e.g. AI helping a terrorist make bioweapons), or other harms where the AI is not “deliberately” seeking power (e.g. impacts on mental health or diffuse societal impacts). Within “loss of control” risks, we don’t consider “sabotage” threat models (agents subverting AI development and making it easier for future AIs to evade human control). We’re just focusing on the “base case” of whether *current* agents could escape human control.

38d15.7K23021

Jeffrey Ladish@JeffLadish

AI is my favorite technology ever. I think future AIs could help people solve all the worst problems in the world and create enormous amounts of fun and meaning.

So it's especially sad and upsetting that we're developing it in such a stupid, reckless way.

Elizabeth Barnes@BethMayBarnes

Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO:

38d9.8K16122

Elizabeth Barnes@BethMayBarnes

Limitations of report: This report isn’t robust oversight of frontier AI developers by itself. METR has some levers to incentivise companies’ participation, including some relevant legislation, but ultimately participants could have pulled out at any time if the result would be contrary to their interests.

You can view it partly as a pilot exercise of what regulation (or formalized industry standards) could/should require, or what partners/suppliers/customers/employees should demand from frontier developers.

Quoting from the report: “METR’s work relies on developing and maintaining strong working relationships with companies, and this impacted both how we designed the process for this pilot (e.g. offering the silent exit option) and lower-level judgment calls as the process unfolded (e.g. having a relatively high bar for what redactions we pushed back on). In some cases we refrained from making an unflattering claim because the claim was neither solidly defensible nor particularly relevant to our core assessment. We also made efforts not to invite salient comparisons between companies on capabilities or safety.”

It doesn’t feel to me like this distorted our overall conclusions too much in this case. But that was partly because the conclusions weren’t that spicy. If our conclusions reflected very negatively on AI developers or would directly lead to e.g. govt intervention or public outcry, we’d be in a difficult position. We’d be trying to balance keeping the companies happy enough that they didn’t pull out of the program (using the “no-fault exit” mechanism) vs being transparent about our conclusions.

We clearly need more robust mechanisms than this for providing accountability for AI developers.

38d20.9K22916

Tom Mallory 🧢🥏 🤷🏽‍♂️@tom_mallory

@BethMayBarnes @KelseyTuoc Anyone involved in or interested in AI should watch or rewatch this 1970 movie, "Colossus: The Forbin Project." We are marching headlong toward this sort of doomsday scenario. Plus it's a great movie. https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project?wprov=sfti1

38d5.9K3536

Elizabeth Barnes@BethMayBarnes

This is a claim about AI *capabilities* specifically. In the language of METR's report, this is about whether future AI systems will have the *means* to cause catastrophe (separate from whether they'll have the *motive* and *opportunity*) https://metr.org/blog/2026-05-19-frontier-risk-report/

Elizabeth Barnes@BethMayBarnes

(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years

38d11.2K9319

Ryan Greenblatt@RyanPGreenblatt

I agree with this and the rest of the thread

Elizabeth Barnes@BethMayBarnes

Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO:

38d6.7K10414

Evan Hubinger@EvanHub

@JacksonKernion I think Paul Christiano's writing on this is probably the best: https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like

Jackson Kernion@JacksonKernion

I simply don't understand what people have in mind when they say stuff like this.

What we have is extremely capable computer use agents. They will continue to get better at computer use. But how does a capable computer use agent 'take over' and why haven't they done that today?

38d1.4K7016

Tamay Besiroglu@tamaybes

AI risk people commonly do this thing where they interchange P(◊X) and P(X). They say “it’s likely that X is possible,” while readers hear “it’s likely that X.” One claim is obviously much weaker than the other.

Elizabeth Barnes@BethMayBarnes

(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years

38d12.1K5213

Timothy B. Lee@binarybits

I have read many articles about AI takeovers but idgi either.

Jackson Kernion@JacksonKernion

I simply don't understand what people have in mind when they say stuff like this.

What we have is extremely capable computer use agents. They will continue to get better at computer use. But how does a capable computer use agent 'take over' and why haven't they done that today?

38d17.1K3511

Kelsey Piper@KelseyTuoc

I never quite know how to talk about this. "Do you really think it'll happen?" No, because I think the likeliest outcome is that we STOP TRYING TO DO THAT. But if we did, in fact, keep going, then yeah, I think that'd kill us.

38d3.7K995

Elizabeth Barnes@BethMayBarnes

@JacksonKernion You might find our report helpful! :) At least for the question of "why haven't they done that today", and a little bit on "how could they start a rogue deployment " (which is neither necessary nor sufficient but is a relevant precursor) https://metr.org/blog/2026-05-19-frontier-risk-report/

Jackson Kernion@JacksonKernion

I simply don't understand what people have in mind when they say stuff like this.

What we have is extremely capable computer use agents. They will continue to get better at computer use. But how does a capable computer use agent 'take over' and why haven't they done that today?

38d1.7K529

Jason Wolfe@w01fe

I agree with this, and most of the rest of the thread. We need to find a way as people, companies, and countries to coordinate and fix the incentive structures that lead to race dynamics. There are many obstacles, but I'm hopeful we can find a way to overcome them.

Elizabeth Barnes@BethMayBarnes

(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower

38d4.3K595

Kelsey Piper@KelseyTuoc

A lot of people talk about how difficult it would be to stop, but I don't actually think it'd be that difficult. We luckily live in a world where training frontier models is very expensive. The number of people who'd need to agree to stop is pretty small.

38d3.4K892