The argument that many would prefer extinction risk to an AI-fueled world government

@eterecursionFor those that don't know, this is the former Chief Scientist at the UK AI Security Institute. Before that, he worked at Google DeepMind, OpenAI, and Google Brain.

@NathanpmYoungA ranty tweet in defence of using numbers to quantify risk: What might happen if there were no AI forecasts? Would people say "wow we don't know what is going to happen with AI, I guess we should be ambivalent between speeding up and slowing down"? No! It would probably (in my view) be full speed ahead. This has forced making poorly justified but precise forecasts to illustrate levels of concern. There are flaws in this, sure, but note that signifcant groups of people in the field don't talk remotely like this about most other things. 1000s of plastic researchers aren't like "there is a 10% chance plastic kills everyone". Oil CEOs are not like "we might end human life". But 1000s of AI researchers do think that. We have polled them, 10% is the median response. So I don't love P(doom) and I would prefer we were clear that 3+ year forecasts are very inaccurate. But let us not pretend that if people had said "I am very worried" this would have been understood as those people intended. They think there is a plausible imminent risk to everyone alive. Some think the risk is likely, some almost certain. I am glad they can convey that, using numbers if they have to.

@MikeIsaaca frustrating thing about all the "TK percent chance we die" headlines is that researchers can say any number and non-technical people will just be like "damn well that guy's in tech, im worried" show your work! is this about scaling laws or some arxiv file ive yet to read?

@NateSilver538The AI risk people actually do show their work! There are hundreds of working papers, thousands of LessWrong posts, literally entire best-selling books about this.

@geoffreyirvingRT @NateSilver538: The AI risk people actually do show their work! There are hundreds of working papers, thousands of LessWrong posts, lite…

@DavidSKruegerRT @NathanpmYoung: Here is a simple idea for AI safety comms people. Say what you think is true. Just be honest.

@NateSilver538I love building empirically grounded statistical models. They work well in data-rich environments like sports. AI x-risk isn't like that. Few technologies are good analogies. Plus, there's survivorship bias, and the track record on nuclear weapons isn't super reassuring, btw.

@DavidSKruegerI've said recently that rogue AI and existential risk are finally getting the attention they deserve from the media. But that's not quite right... We're talking about the end of humanity. We should have round the clock coverage until the situation is resolved.

@tristanharrisRT @eterecursion: For those that don't know, this is the former Chief Scientist at the UK AI Security Institute. Before that, he worked at…

@krishnanrohitWhen you're talking about doom, you should make it exceedingly clear that the chain of events is: 1. Fable 5.2 -> 2. Fable 7 -> 3. SuperDangerousASI 4. Doom And that you assume this will happen spontaneously and over v short timeframe (days/weeks). Otherwise you're misleading.

@DavidSKruegerWhen I talk to people in AI safety about how they think the worlds where we *don't* all die go, it's usually some combination of: 1) We build a superintelligent AI that takes over the world, but it's nice to us (I don't expect this to bereassuring to most people). 2) The world gets it's shit together and stops letting us build dangerous shit. Significantly, *the societal response is built in*.

@j_foerstPrediction markets usually break down for existential risks - e.g. if everyone gets wiped out by a comet then nobody is there to win the bet (and therefore nobody will take the other side). P(doom) is different: once there is an entity meanfigfully taking the other side of the bet that becomes informative in its own right: https://manifold.markets/MartinRandall/will-ai-wipe-out-humanity-before-th-d8733b2114a8

@beenwrektTed Chiang wrote the definitive essay on p(doom) in 2017. It's worth reading again in light of this week's events. https://www.buzzfeednews.com/article/tedchiang/the-real-danger-to-civilization-isnt-ai-its-runaway

@Grady_BoochI assert that even if there is no slowdown - coordinated or not - the prediction of a 70% chance of human extinction is complete and utter nonsense. Yes there are clear and present harms, and this doomerism is a dangerous distraction from the real harms about which we can take action.

@RichardSocherReally important and timely conversation around the safety of recursive self-improving superintelligence. Extraordinary claims of extinction probabilities require extraordinary evidence not extrapolation plus handwaving about progress. Thanks @nxthompson for asking the hard and interesting questions. https://www.youtube.com/watch?v=eqYqUAdjsL0

@GaryMarcusp(doom) < p(catastrophe) < p(dystopia)

@paularamblesanthropic when you say p(doom) < 5%

@DeryaTR_One of the greatest intellectual dishonesties of the AI age is assigning an “X% chance” that AI will kill everyone and presenting that number as science! 10%? 20%? 50%? Show me the damn data! Show me the model. Show how you calculated it! Instead, we get chains of speculative assumptions and science-fiction scenarios, including implausible claims about engineered pandemics. I have worked with viruses for decades; biology does not work like a doomsday screenplay. The burden of proof belongs to the person making the extraordinary claim. Nobody else has to prove the probability is zero. If I claimed there was a 10% chance aliens would arrive within ten years and destroy Earth, you would ask where the 10% came from. Possibility and probability are not the same thing. It may be possible that there are million intelligent civilization in our galaxy or none, there is no probability you can assign to that without making wild assumptions. Ask the same of AI extinction. A subjective fear does neither become science because you attach a percentage to it nor make you a credible expert!

@beffjezosRT @zackkanter: So many people on the timeline can’t entertain the reality that for many people, a 10% chance of ending humanity is far pre…

@charles_irlCome all you MoTS and listen to me: Just lengthen that timeline and let that p(doom) be!

@firstadopterThe past week has been shameful and irresponsible for everyone involved. Read this: "this week, ex-Anthropic employee Jacob Coxon claimed — and then Anthropic Alignment Science lead Evan Hubinger agreed — that the probability that AI will "kill all humans" is ">10% in the next decade" (!!). That is, the claim is not merely that AI could kill thousands, millions, or even billions of humans (claims that themselves would be extraordinary!), but that there is a greater than ten percent chance that AI will kill all humans. That is, if you have a newborn, their claim is that there is a greater than ten percent chance that your child will somehow perish at the hands of AI before they enter middle school. These ghoulish claims strike brazenly at the hearth, and given the obvious importance of AI, it is unsurprising that they have leapt into the mainstream, with people asking the natural question: how would that happen? The answers always rely on hand-wavy extrapolation into the future; for example, Jacob Coxon cites "hacking critical infrastructure" and "extinction-level bioweapons" without further elaboration. But Coxon is not an expert on critical infrastructure, nor on bioweapons — nor, for that matter, on extinction. What 27-year-old Coxon is an expert on — if accidentally — is the contagion of fear." "We should heed the wisdom of the late Carl Sagan that extraordinary claims require extraordinary evidence: the claims from Coxon and his ilk are the most extraordinary a technologist can make, and we must demand evidence commensurate with the claims." https://bcantrill.dtrace.org/2026/09/13/the-contagion-of-fear/

@ylecunRT @DeryaTR_: One of the greatest intellectual dishonesties of the AI age is assigning an “X% chance” that AI will kill everyone and presen…

@nxthompsonRT @RichardSocher: Really important and timely conversation around the safety of recursive self-improving superintelligence. Extraordinary…

@ajeya_cotraRT @NateSilver538: I love building empirically grounded statistical models. They work well in data-rich environments like sports. AI x-risk…

@GaryMarcusAI doomers - do you actually think we will be extinct in five years or is that just what Jacob Coxon is telling the media?

@adamjohnsonCHIWithout getting into the weeds of the technical aspects, I feel like—high level— we should be having more of a discussion about how many of the go-to experts on AI extermination, namely this one cited several times in the NYT, are completely bat shit EA fantasists.

@krishnanrohitEvery single one of these, and thousands besides, are all asking the same ur-question, which is that "foom" is just not a thing that can plausibly happen. Every major doom prediction hinges on foom, since all other problems can be solved or played through.

@DavidSKruegerPeople need to see this. Experts say 18% chance of extinction-level outcomes from AI. Totally insane that we have taken things this far, to the point where we literally have rogue AI swarms on the internet. There's still time, but we need to stop now, before it's too late.

@KatjaGraceI’m an independent researcher who runs the largest and longest-running peer-reviewed survey of AI researchers. These researchers are experts who publish papers at the top machine learning and artificial intelligence venues.

@KatjaGraceWe’ve been conducting these surveys since 2016, so we can see how expert opinion has moved over the years. Two trends: 1) Researchers' 'timelines' to human-level AI have shrunk, fast. 2) Most researchers have seen major future risks of extinction-level catastrophes for a long time, and their concerns are lately trending upward

@KatjaGrace(A figure of how the distribution of concern changed in the year before this survey, that I'm told is too complicated for people, but give it a go!)

@stevesiRT @ramez: I love this. My p(doom) was always low. It's gotten even lower all through the last month. Early malfunctions that 1. Don't c…

@chamathRT @EQworksinBiz: @chamath "They shouldn't. It is made up". Six words, put the whole 10% probability of extinction by AI to bed.

@martin_casadoJust spitballing here ... but maybe because milquetoast measures are not how you handle something with a greater than 10% chance of killing all humans. And the person who proposed it agrees more than he disagrees that it has a 10% change of killing all humans ... ?

@DaniloJRezendeRT @SchmidhuberAI: Does your company need more attention or investment? Just tell the world that there's a 10% chance your product will kil…

@merriermHowever, in my opinion, conflating "vibes-based" estimates with empirically validated risk likelihoods is not a responsible approach to broaching important AI Safety discussions with the media, policymakers, or the general public.

@beenwrektp(earthquake) as an analogy for p(doom). https://www.argmin.net/p/the-chances-of-earthquakes

@binarybits@MaxNadeau_ Do you have a suggestion for a neutral label for someone who thinks there is a significant chance AI will kill everyone due to superintelligence/loss of control? I suggested "singularist" in a piece three years ago but that hasn't caught on.

@suchenzangall you kids don't realize they must have a scaling law for p(doom) (top secret, too dangerous of a plot to release) it is definitely exponential! not sigmoid! all the points on the log log log log plot says so!

@nxthompson12-year-old: “Do the people at the AI companies really think the world is going to end in three years?” Me: “some of them. But it’s more like five years.” 12-year-old: “phew. That’s better. I’ve always wanted to turn 17.”

@HeidyKhlaafIt was great to speak to the @guardian alongside many other critical voices (e.g., @ProfWoodward and @GaryMarcus) about the unsubstantiated AI extinction claims being made, that the public is being compelled to believe with no evidence.

@GaryMarcusRT @HeidyKhlaaf: It was great to speak to the @guardian alongside many other critical voices (e.g., @ProfWoodward and @GaryMarcus) about th…

@martin_casadoPacing and light oversight is clearly the correct response for *checks notes* 70% chance of globally catastrophic outcomes from AI.

@jasonlkThe 10% P(doom) stuff is overwrought. If the feds thought anyone in downtown San Francisco was building tech with a 10% chance of blowing up the world in ten years, they would move in with a SWAT team and close it down. They have not. The government is looking at it and going: this is a bunch of excited teenagers. We will step in later if it gets crazy.

@bradrcarsonI'm just a half-failed politician from small-town Oklahoma, and not a Princeton or UMass academic, but this isn't right. As much as I admire the AI as Normal Technology guys, this article and tweet are, IMO, way way off. To wit, their article accepts subjective probability in principle (their words) but they then build their case on the claim that a credence without a reference class or a model is just “feelings dressed up as numbers.” Those can't both be right. As I read them, the P(doom) people are working in a tradition, from Ramsey through Jaynes (super important to this group esp Yudkowsky), in which every probability is a degree of belief and a reference class is just one kind of evidence. This to be clear isn’t my take, as I’m more of a Knightian (who a lot of the rationalists disdain, I think) and Elster (who is never cited). But the “Normal Tech” critique has to, well, meet the tradition it's criticizing. I don’t think they do that in this article. Say what you want to about the credence approach, but some of the most important thinkers on statistics built the edifice of it. If you want to say Jaynes, Ramsey, de Finetti, Savage, or Cox et al are wrong, make the case. But it’s not crazy to value them. They are very important thinkers, IMO, and it is not fraudulent to adopt their methods, even if you think it's ultimately wrong. Even on the empirics, giving numbers allows a debate, and as the rats say, you have a number even if you don't offer one (see their criticism of Knight, alas). @random_walker @itaisher

@xuanalogueReminded by @itaisher's critique of giving P(doom) numbers w/o public justification of the following idea: Bayesian beliefs are beliefs we'd bet on, but perhaps the beliefs we should use for public decision-making are the beliefs we can justify to others.

The argument that many would prefer extinction risk to an AI-fueled world government

TLDR

Combined views

The argument that many would prefer extinction risk to an AI-fueled world government

46 Sources

Combined views

Sentiment

Summary

TLDR

Sentiment

Summary

Sentiment

Summary

Sentiment

Summary

46 Sources