• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    How could AI kill everyone? A post highlights five risk scenarios

    The author recommends AI 2027 as realistic and highly detailed, arguing that prominent AI safety researchers have put substantial work into concrete accounts of catastrophic risk.

    DK
    GI
    RS
    13 Sources, ,

    TLDR

    A post disputes the claim that AI safety researchers have no realistic answer to how AI could kill everyone. Its recommended reading includes AI 2027, scenarios by Paul Christiano, Gwern Branwen and Joshua Clymer, and a high-level explanation by Holden Karnofsky. The author also highlights AI 2040 and AI 2027 for the effort devoted to developing detailed risk scenarios.

    Combined views

    459.6K

    13 Sources, first seen 19d ago

    Combined views

    459.6K

    13 Sources, first seen 19d ago

    3K likes
    19d ago
    first seen 19d ago
    3K likes
    168 comments
    3.2K saves
    1K reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    168 comments
    3.2K saves
    1K reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    13 Sources

    @S_OhEigeartaighThis is a very helpful public service from Zvi: collecting all the statements about AI Xrisk concerns in industry being real/shared/endorsed: https://thezvi.substack.com/p/the-extinction-risk-preference-cascade
    @ohabrykaThere are about 5-10 different in-depth scenarios people have written over the years. AI-2027 is the easiest one to point to: https://ai-2027.com/ There is a good scenario in http://ifanyonebuilds.it Here is Paul Christiano: https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like Here is Gwern Branwen: https://www.lesswrong.com/posts/a5e9arCnbDac9Doig/it-looks-like-you-re-trying-to-take-over-the-world
    @DavidSKruegerRT @TheZvi: I'm trying to catch everyone, but starting a thread for anyone at OpenAI or Anthropic (or other labs) that wants to ensure thei…
    @RosieCampbellRT @ohabryka: @NathanJRobinson There are about 5-10 different in-depth scenarios people have written over the years. AI-2027 is the easie…
    @geoffreyirvingRT @ohabryka: After Jacob Coxon's resignation and extinction warnings, a lot of people are asking 'how could AI possibly kill everyone?' an…
    @krishnanrohitThis is a good list, however these scenarios are detailed, but none are comparable to how, eg, climate science discussed its scenarios. These are not actually useful models. It's not about numbers vs text (you can guesswork numbers), or even about models vs prose (same reason). - AI 2027 models takeoff based on AI research automation and assumes it into other acceleration incl estimates of capability milestones. It doesn't model the reactions from the world - Paul's scenario has broad mechanisms, useful conceptually, but it does not show a mechanism - Gwern - great catastrophe fiction, prob the best here. It's a scenario, sure, but not a model, though it's nice that its technically specific - Karnofskys argument is interesting, and it supports maybe some premises of human disempowerment *if* the assumptions are true (big if, like AI being hostile to humans) - Clymer's writing even he says is nota. prediction and is highly uncertain. Not empirical or a constrained extinction model And even in these scenarios, most of them don't model the world's reactions, nor do they model the various lynch points where the plan might shift, nor why/ what/ when one ought to worry (beyond just now, preemptively). And *even then*, even if all these are plausible, you still need to show why you can't just shut those pathways down instead of shutting down the entire progress vector. "This is one plausible path, there are hundreds" is usually not a good enough reason, the same way "computers can be hacked, lets never connect them together" would not be a good reason. You have to demonstrate the complete inability to control through normal means. That's what a model discussing the problem has to show. It is very difficult to show this. Because here the feedback loops are complicated. Even for complex systems like the climate which have tons of feedback loops which are modellable in principle struggle with this. But that's okay! It should be hard. I suspect Oliver sees these as "you asked us for models, we have plenty of models", vs folks going "those are not models, most of the itneresting parts are the assumptions" and "you haven't said even if that pathway is true why that necessitates shutting down the program vs just stopping that pathway". These are decent pathways of how x-risk could happen, but they mix up stories and conceptual leaps and past catastrophes and governance proposals. It's a grabbag, but but none seem insurmountable, nor defined well enough that we know the right course of action, being to solve the problems identified vs give up.
    @sarthakgh.@TheZvi has compiled a list of people across labs coming out in favor of slowing down in the last few days: https://thezvi.substack.com/p/the-extinction-risk-preference-cascade
    @ScobleizerRT @robleclerc: if this is the best they’ve got, i’ll be downgrading my p(doom) even further. all these scenarios describe TOTAL and inte…
    @lukeprogRT @ohabryka: After Jacob Coxon's resignation and extinction warnings, a lot of people are asking 'how could AI possibly kill everyone?' an…
    @beffjezosPseudoscientific cottage industry of armchair philosophers looking to secure a permanent lucrative position of soft power as we achieve AGI Their future paychecks depend on them dooming, of course they will say whatever they need to to engineer this reg capture outcome.

    13 Sources

    @S_OhEigeartaighThis is a very helpful public service from Zvi: collecting all the statements about AI Xrisk concerns in industry being real/shared/endorsed: https://thezvi.substack.com/p/the-extinction-risk-preference-cascade
    @ohabrykaThere are about 5-10 different in-depth scenarios people have written over the years. AI-2027 is the easiest one to point to: https://ai-2027.com/ There is a good scenario in http://ifanyonebuilds.it Here is Paul Christiano: https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like Here is Gwern Branwen: https://www.lesswrong.com/posts/a5e9arCnbDac9Doig/it-looks-like-you-re-trying-to-take-over-the-world
    @DavidSKruegerRT @TheZvi: I'm trying to catch everyone, but starting a thread for anyone at OpenAI or Anthropic (or other labs) that wants to ensure thei…
    @RosieCampbellRT @ohabryka: @NathanJRobinson There are about 5-10 different in-depth scenarios people have written over the years. AI-2027 is the easie…
    @geoffreyirvingRT @ohabryka: After Jacob Coxon's resignation and extinction warnings, a lot of people are asking 'how could AI possibly kill everyone?' an…
    @krishnanrohitThis is a good list, however these scenarios are detailed, but none are comparable to how, eg, climate science discussed its scenarios. These are not actually useful models. It's not about numbers vs text (you can guesswork numbers), or even about models vs prose (same reason). - AI 2027 models takeoff based on AI research automation and assumes it into other acceleration incl estimates of capability milestones. It doesn't model the reactions from the world - Paul's scenario has broad mechanisms, useful conceptually, but it does not show a mechanism - Gwern - great catastrophe fiction, prob the best here. It's a scenario, sure, but not a model, though it's nice that its technically specific - Karnofskys argument is interesting, and it supports maybe some premises of human disempowerment *if* the assumptions are true (big if, like AI being hostile to humans) - Clymer's writing even he says is nota. prediction and is highly uncertain. Not empirical or a constrained extinction model And even in these scenarios, most of them don't model the world's reactions, nor do they model the various lynch points where the plan might shift, nor why/ what/ when one ought to worry (beyond just now, preemptively). And *even then*, even if all these are plausible, you still need to show why you can't just shut those pathways down instead of shutting down the entire progress vector. "This is one plausible path, there are hundreds" is usually not a good enough reason, the same way "computers can be hacked, lets never connect them together" would not be a good reason. You have to demonstrate the complete inability to control through normal means. That's what a model discussing the problem has to show. It is very difficult to show this. Because here the feedback loops are complicated. Even for complex systems like the climate which have tons of feedback loops which are modellable in principle struggle with this. But that's okay! It should be hard. I suspect Oliver sees these as "you asked us for models, we have plenty of models", vs folks going "those are not models, most of the itneresting parts are the assumptions" and "you haven't said even if that pathway is true why that necessitates shutting down the program vs just stopping that pathway". These are decent pathways of how x-risk could happen, but they mix up stories and conceptual leaps and past catastrophes and governance proposals. It's a grabbag, but but none seem insurmountable, nor defined well enough that we know the right course of action, being to solve the problems identified vs give up.
    @sarthakgh.@TheZvi has compiled a list of people across labs coming out in favor of slowing down in the last few days: https://thezvi.substack.com/p/the-extinction-risk-preference-cascade
    @ScobleizerRT @robleclerc: if this is the best they’ve got, i’ll be downgrading my p(doom) even further. all these scenarios describe TOTAL and inte…
    @lukeprogRT @ohabryka: After Jacob Coxon's resignation and extinction warnings, a lot of people are asking 'how could AI possibly kill everyone?' an…
    @beffjezosPseudoscientific cottage industry of armchair philosophers looking to secure a permanent lucrative position of soft power as we achieve AGI Their future paychecks depend on them dooming, of course they will say whatever they need to to engineer this reg capture outcome.