• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    Noah Smith Shares AI Supervirus Extinction Scenario

    Eliezer Yudkowsky disputes Noah Smith's jailbroken LLM biolab hypothesis.

    TB
    RT
    3 Sources, 33d ago, first seen 33d ago

    TLDR

    Noah Smith posted that a jailbroken LLM enabling an East European biolab to engineer a deadly virus represents the most likely path for AI to kill humanity. Timothy B. Lee noted the post and highlighted Eliezer Yudkowsky's reply rejecting that scenario. The exchange centers on whether current AI tools could realistically accelerate high-lethality pathogen design leading to societal collapse. No independent confirmation of the hypothetical's details appears in the posts.

    Combined views

    53K

    3 Sources, first seen 33d ago

    Combined views

    53K

    3 Sources, first seen 33d ago

    352 likes
    352 likes
    20 comments
    100 saves
    29 reposts
    20 comments
    100 saves
    29 reposts

    Sentiment

    Positive23%77%Negative

    Summary

    Sentiment

    Positive23%77%Negative

    Replies largely dismissed claims of AI-designed superviruses as the top existential risk, calling the scenarios stupid or unrealistic, while positive accounts praised the biologist’s assessment as authoritative and worth expanding.

    Based on 70 sentiment-bearing replies from 61 accounts across 3 conversations.

    Summary

    Replies largely dismissed claims of AI-designed superviruses as the top existential risk, calling the scenarios stupid or unrealistic, while positive accounts praised the biologist’s assessment as authoritative and worth expanding.

    Based on 70 sentiment-bearing replies from 61 accounts across 3 conversations.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    3 Sources

    @NoahpinionIf AI does kill the human race, I think this is the most likely scenario for how it happens.
    @binarybitsYud accepts no substitutes.
    @RuxandraTesloI think this post somewhat overstates the risk from AI-designed viruses, although I agree there are real risks and we should prepare for them. So I wrote an explainer of what I think the risks are. At the start, I will make a distinction between 2 scenarios: AI helping people use or distribute pathogens that already exist, and AI enabling the design of novel pathogens or "hyper-viruses". The first risk is that AI will lower barriers to entry for a large number of people to use and distribute existing pathogens. So the AI would help a not very sophisticated but malign actor identify a dangerous pathogen, understand how to handle it, or optimize its dissemination. This is the “make a virus in your garage laboratory scenario” where individual, rogue actors can inflict substantial damage. First, I’d say that it’s not quite do-able in a lab garage and I think it will require more tacit knowledge and access to resources than many imagine. But still, it is plausible in the near and medium. This scenario is concerning, but I think it’s important to note that its ultimate harm is also constrained by the fact that existing pathogens are, by definition, already known to us. As a result, for most of them, we have some combination of prior immunity, scientific understanding, diagnostics and treatments. There are, of course, important exceptions to this. One of them is smallpox. Yet even for smallpox, large vaccine stockpiles exist. We can manufacture additional doses, should it be needed. So overall, I think that an attack involving an existing pathogen could be very damaging (the extent to which would depend on our ability to respond; more on that later), but not civilization ending. Now moving on to the next scenario, which is more frightening and which most people refer to when they talk about AI-designed viruses. The idea here is that AI would design a genuinely novel pathogen that combines high transmissibility with extreme lethality or other unusually dangerous properties that take a long time to co-evolve in nature. I strongly believe this will remain very difficult to engineer for the foreseeable future and that the capabilities required, in terms of personnel, funding and access to laboratories will be substantial. The main reason is that biology is a highly empirical discipline, that one cannot engineer and predict simply by reading enough internet text. In other words, you need the relevant pathogen-specific experimental data, much of which does not currently exist. Not only that: you need continuous feedback loops of evolving that virus in a lab and learning from it. The thing is, relatively few organizations are equipped to do this. This is for the foreseeable future more like a state level effort rather than a few rogue scientists in their backyard. And even then the effort would be expensive, slow and I think it would also leave traces, as it would require specialized facilities, trained staff and sustained experimentation. So this is not at all a rogue individual scenario, but would require a pretty expensive and “loud” concerted effort. At some point, perhaps decades from now, someone may build the experimental infrastructure and datasets required to train systems that are genuinely capable of zero-shot “hyper-virus” design. Now, let’s suppose this happens. That is scary! But if it does, I think it's a mistake to discuss this possibility as though offensive biotechnology will advance while defensive biotechnology will remain frozen. If we will be able to quickly design hyper-viruses, we will also be able to quickly design defensive therapeutics against them. And there are reasons to believe the defensive problem is in important ways easier. We have created a large number of vaccines and other therapeutics. Our prior should be that If AI becomes dramatically better at zero-shot biological design, some of its earliest and most reliable successes will be on the defensive side. So in my mind, the strategy should in large part be offensive-defensive. Rather than trying only to prevent every imaginable pathogen from ever being created, we should invest much more heavily in defenses that work across many pathogens at once. Now, to the point of distribution: it is true that in any of these scenarios, how fast we respond in terms of actually manufacturing and distributing the virus will determine how damaging the attack would be. This is why I support measures like building much more latent manufacturing capacity for vaccines and therapeutics, so that new countermeasures can be produced quickly when a threat appears. In my view, this is a much more tractable engineering problem than guaranteeing that no dangerous pathogen will ever emerge, and platforms such as RNA therapeutics should make flexible manufacturing increasingly feasible. Historically, the world has invested relatively little in this kind of infrastructure. This is because respiratory infection has not been treated as an especially large problem in the overall scheme of things. If the perceived threat from engineered pandemics rises, this should change (and that would be good).

    3 Sources

    @NoahpinionIf AI does kill the human race, I think this is the most likely scenario for how it happens.
    @binarybitsYud accepts no substitutes.
    @RuxandraTesloI think this post somewhat overstates the risk from AI-designed viruses, although I agree there are real risks and we should prepare for them. So I wrote an explainer of what I think the risks are. At the start, I will make a distinction between 2 scenarios: AI helping people use or distribute pathogens that already exist, and AI enabling the design of novel pathogens or "hyper-viruses". The first risk is that AI will lower barriers to entry for a large number of people to use and distribute existing pathogens. So the AI would help a not very sophisticated but malign actor identify a dangerous pathogen, understand how to handle it, or optimize its dissemination. This is the “make a virus in your garage laboratory scenario” where individual, rogue actors can inflict substantial damage. First, I’d say that it’s not quite do-able in a lab garage and I think it will require more tacit knowledge and access to resources than many imagine. But still, it is plausible in the near and medium. This scenario is concerning, but I think it’s important to note that its ultimate harm is also constrained by the fact that existing pathogens are, by definition, already known to us. As a result, for most of them, we have some combination of prior immunity, scientific understanding, diagnostics and treatments. There are, of course, important exceptions to this. One of them is smallpox. Yet even for smallpox, large vaccine stockpiles exist. We can manufacture additional doses, should it be needed. So overall, I think that an attack involving an existing pathogen could be very damaging (the extent to which would depend on our ability to respond; more on that later), but not civilization ending. Now moving on to the next scenario, which is more frightening and which most people refer to when they talk about AI-designed viruses. The idea here is that AI would design a genuinely novel pathogen that combines high transmissibility with extreme lethality or other unusually dangerous properties that take a long time to co-evolve in nature. I strongly believe this will remain very difficult to engineer for the foreseeable future and that the capabilities required, in terms of personnel, funding and access to laboratories will be substantial. The main reason is that biology is a highly empirical discipline, that one cannot engineer and predict simply by reading enough internet text. In other words, you need the relevant pathogen-specific experimental data, much of which does not currently exist. Not only that: you need continuous feedback loops of evolving that virus in a lab and learning from it. The thing is, relatively few organizations are equipped to do this. This is for the foreseeable future more like a state level effort rather than a few rogue scientists in their backyard. And even then the effort would be expensive, slow and I think it would also leave traces, as it would require specialized facilities, trained staff and sustained experimentation. So this is not at all a rogue individual scenario, but would require a pretty expensive and “loud” concerted effort. At some point, perhaps decades from now, someone may build the experimental infrastructure and datasets required to train systems that are genuinely capable of zero-shot “hyper-virus” design. Now, let’s suppose this happens. That is scary! But if it does, I think it's a mistake to discuss this possibility as though offensive biotechnology will advance while defensive biotechnology will remain frozen. If we will be able to quickly design hyper-viruses, we will also be able to quickly design defensive therapeutics against them. And there are reasons to believe the defensive problem is in important ways easier. We have created a large number of vaccines and other therapeutics. Our prior should be that If AI becomes dramatically better at zero-shot biological design, some of its earliest and most reliable successes will be on the defensive side. So in my mind, the strategy should in large part be offensive-defensive. Rather than trying only to prevent every imaginable pathogen from ever being created, we should invest much more heavily in defenses that work across many pathogens at once. Now, to the point of distribution: it is true that in any of these scenarios, how fast we respond in terms of actually manufacturing and distributing the virus will determine how damaging the attack would be. This is why I support measures like building much more latent manufacturing capacity for vaccines and therapeutics, so that new countermeasures can be produced quickly when a threat appears. In my view, this is a much more tractable engineering problem than guaranteeing that no dangerous pathogen will ever emerge, and platforms such as RNA therapeutics should make flexible manufacturing increasingly feasible. Historically, the world has invested relatively little in this kind of infrastructure. This is because respiratory infection has not been treated as an especially large problem in the overall scheme of things. If the perceived threat from engineered pandemics rises, this should change (and that would be good).