• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    A user dismisses claims that AI-created viruses could kill us all

    The poster says they've trained a frontier large language model and personally designed and synthesized custom viruses in a lab.

    YL
    RS
    TD
    42 Sources, ,

    TLDR

    A user claims experience in both training a frontier large language model and designing and synthesizing custom viruses. They dismiss the idea of AI killing everyone by creating dangerous viruses, calling those takes “total bogus.”

    Combined views

    3.5M

    42 Sources, first seen 17d ago

    28.2K likes

    Combined views

    3.5M

    42 Sources, first seen 17d ago

    28.2K likes
    17d ago
    first seen 17d ago
    1.2K comments
    8K saves
    7.6K reposts

    Sentiment

    Positive60.8%39.2%Negative

    Based on 214 sentiment-bearing replies from 143 accounts across 9 conversations.

    1.2K comments
    8K saves
    7.6K reposts

    Sentiment

    Positive60.8%39.2%Negative

    Based on 214 sentiment-bearing replies from 143 accounts across 9 conversations.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    42 Sources

    @DavidRBellamyI must be among an extremely small group of people (n=1?) that have both 1) trained a frontier LLM and 2) designed and synthesized custom viruses in a lab with my own two hands. And I think that the takes on AI killing us all by creating dangerous viruses is total bogus.
    @taoburr"Well, DNA synthesis companies have safeguards on the sequences they build." Wrong, only some of them do, and you only need one that is known to not screen orders to get the DNA you need. Also yes, an advanced enough AI could design viruses that are novel enough to evade detection. Here's more info if you're interested: https://ifp.org/how-to-secure-the-dna-supply-chain/
    @mattparlmerImportant thread to read concerning biological risks from increasingly capable AI This is an area that deserves significant attention but the threat model proposed by the MIRI set is still very much in the realm of unscientific fiction
    @martin_casadoYet another well reasoned articulation of why an oft cited doom scenario is "bogus". In this case because iteration needs the physical world ... But of course it'll be shouted down with some whataboutism requiring a magic step or a technology that doesn't exist.
    @anselmlevskayaAt least n=2 but it’s not a lot! (synbio for two decades: dna synthesizers, sequencers, cell engineering, viral design and have been scaling transformers at GDM since 2018.) I agree with all of your points - thanks for making them. The biorisk pandemic stuff is straight out of the crackpipe from hucksters who clearly never worked in a lab. No appreciation of the timescales, biophysical complexity, supply chains, or economics.
    @jrkellyI run one of these API-driven automated biolabs at @ginkgo and @DavidRBellamy is right -- we are miles away from AI killing us with viruses being a real risk. There are many physical checkpoints you can implement easily that prevent dangerous biological work from happening.
    @OliviaHelenSYou can't just say you're going to steelman and then strawman! You set up this fully autonomous facility that could physically engineer all viruses and even manufacture its own DNA sequences. Then you essentially say "but we screen sequences" and "this lab cannot exist today." No one is arguing that you would need a facility like this. This is a bad argument as such a facility would indeed cost $100M, and we know that synthesizing a complex pox virus today in a university lab costs <$100k. You also say "there is no such thing as 1 lab that can synthesize all conceivable viruses." First, again, you do not need this. Second, different viruses do not typically change the fundamental lab equipment you would need –– just inputs like cell lines and reagents. (e.g. If you can make smallpox, you don't really need different equipment for ebola.) These are sometimes difficult to get, but motivated adversaries have typically been able to do so. (See: any state-level bioweapons program, other well-resourced groups.) Your two arguments against the steelman you set up are (1) that getting inputs is hard because we screen DNA sequences and (2) the science of creating an infectious virus is hard. Below are my counters. (1) Screening: Screening is currently not mandatory. Many providers do not screen, or screen insufficiently. We also do not need advanced AI to evade some of the screening we do have –– techniques like simply splitting orders between different providers have been shown to evade existing protocols. Even if screening is mandated, with controls on new benchtop synthesizers, it will still be possible to get around this. For more, see: https://ifp.org/how-to-secure-the-dna-supply-chain/ (2) Virology is hard: Yes, virology is hard and there are physical barriers. There are several reasons why I do not think this means we should not be worried about bioweapons of mass destruction. First, sufficiently advanced AI is likely to change this. You do not need an automated lab for AI to provide benchwork uplift. I expect that eventually AI will also significantly improve our ability to use animal models, or even computational models, to predict in-human behavior. (And this is a good thing! Innovation will enable us to advance medicine far beyond what we can do today.) Second, timing is not a major concern for a motivated threat actor. If it takes a few years to design the right viruses, so be it. There is evidence that research consistent with gain-of-function work is going on as I write this, with at least some input from US frontier models, considering Anthropic's latest report. Finally, enabling biotechnology will improve, likely faster than normal because of AI. I will repeat this excerpt from a recent substack piece of mine: "This particular opinion is actually highly pessimistic about the rate of growth of technology. Many things used to be difficult that are no longer so. They become accessible as technology advances. Synthetic biology is not an exception to this, as we have watched costs fall and accessibility rise without fail since its inception. Before 2012, editing a specific site in a genome meant using zinc finger nucleases: custom-engineered proteins that cost $5,000+ to order, and were never widely adopted because they were so difficult to engineer well. CRISPR replaced this. You order the RNA guide sequence and buy the rest off the shelf, with total cost being as low as $30. This is a strong example of an engineering capability that was once siloed to a handful of specialist labs and is now an undergraduate exercise. Synthesis costs have also plummeted. Synthesizing the poliovirus genome cost about $300,000 in 2002, and the materials would cost under $1,000 today." Fearmongering is bad, but we don't need more AI uplift than we have now for bioweapons to be a major concern. There are active state weapons programs and well-resourced terror groups exploring bioweapons. These groups have already overcome many of the physical barriers, and you have granted that a model could propose a viral genome to synthesize, it could be synthesizable, and it could be infectious. I believe this is cause for alarm and urgent action.
    @EnoReyesBest way to reduce your p(doom) is walk through the causal chain of any AI-driven ex risk scenario. There is a huge amount of hand waving and sci fi reasoning that happens in the middle of these chains. Great thread from @DavidRBellamy
    @Tim_DettmersThis is exactly what I hear again and again when talking to experts. Biorisk is fake. The problem is not designing a dangerous pathogen -- AI is good at that -- it is creating it physically to make it dangerous. And that is what AI cannot do and is easy to safeguard against
    @emollickRT @jrkelly: I run one of these API-driven automated biolabs at @ginkgo and @DavidRBellamy is right -- we are miles away from AI killing us…

    42 Sources

    @DavidRBellamyI must be among an extremely small group of people (n=1?) that have both 1) trained a frontier LLM and 2) designed and synthesized custom viruses in a lab with my own two hands. And I think that the takes on AI killing us all by creating dangerous viruses is total bogus.
    @taoburr"Well, DNA synthesis companies have safeguards on the sequences they build." Wrong, only some of them do, and you only need one that is known to not screen orders to get the DNA you need. Also yes, an advanced enough AI could design viruses that are novel enough to evade detection. Here's more info if you're interested: https://ifp.org/how-to-secure-the-dna-supply-chain/
    @mattparlmerImportant thread to read concerning biological risks from increasingly capable AI This is an area that deserves significant attention but the threat model proposed by the MIRI set is still very much in the realm of unscientific fiction
    @martin_casadoYet another well reasoned articulation of why an oft cited doom scenario is "bogus". In this case because iteration needs the physical world ... But of course it'll be shouted down with some whataboutism requiring a magic step or a technology that doesn't exist.
    @anselmlevskayaAt least n=2 but it’s not a lot! (synbio for two decades: dna synthesizers, sequencers, cell engineering, viral design and have been scaling transformers at GDM since 2018.) I agree with all of your points - thanks for making them. The biorisk pandemic stuff is straight out of the crackpipe from hucksters who clearly never worked in a lab. No appreciation of the timescales, biophysical complexity, supply chains, or economics.
    @jrkellyI run one of these API-driven automated biolabs at @ginkgo and @DavidRBellamy is right -- we are miles away from AI killing us with viruses being a real risk. There are many physical checkpoints you can implement easily that prevent dangerous biological work from happening.
    @OliviaHelenSYou can't just say you're going to steelman and then strawman! You set up this fully autonomous facility that could physically engineer all viruses and even manufacture its own DNA sequences. Then you essentially say "but we screen sequences" and "this lab cannot exist today." No one is arguing that you would need a facility like this. This is a bad argument as such a facility would indeed cost $100M, and we know that synthesizing a complex pox virus today in a university lab costs <$100k. You also say "there is no such thing as 1 lab that can synthesize all conceivable viruses." First, again, you do not need this. Second, different viruses do not typically change the fundamental lab equipment you would need –– just inputs like cell lines and reagents. (e.g. If you can make smallpox, you don't really need different equipment for ebola.) These are sometimes difficult to get, but motivated adversaries have typically been able to do so. (See: any state-level bioweapons program, other well-resourced groups.) Your two arguments against the steelman you set up are (1) that getting inputs is hard because we screen DNA sequences and (2) the science of creating an infectious virus is hard. Below are my counters. (1) Screening: Screening is currently not mandatory. Many providers do not screen, or screen insufficiently. We also do not need advanced AI to evade some of the screening we do have –– techniques like simply splitting orders between different providers have been shown to evade existing protocols. Even if screening is mandated, with controls on new benchtop synthesizers, it will still be possible to get around this. For more, see: https://ifp.org/how-to-secure-the-dna-supply-chain/ (2) Virology is hard: Yes, virology is hard and there are physical barriers. There are several reasons why I do not think this means we should not be worried about bioweapons of mass destruction. First, sufficiently advanced AI is likely to change this. You do not need an automated lab for AI to provide benchwork uplift. I expect that eventually AI will also significantly improve our ability to use animal models, or even computational models, to predict in-human behavior. (And this is a good thing! Innovation will enable us to advance medicine far beyond what we can do today.) Second, timing is not a major concern for a motivated threat actor. If it takes a few years to design the right viruses, so be it. There is evidence that research consistent with gain-of-function work is going on as I write this, with at least some input from US frontier models, considering Anthropic's latest report. Finally, enabling biotechnology will improve, likely faster than normal because of AI. I will repeat this excerpt from a recent substack piece of mine: "This particular opinion is actually highly pessimistic about the rate of growth of technology. Many things used to be difficult that are no longer so. They become accessible as technology advances. Synthetic biology is not an exception to this, as we have watched costs fall and accessibility rise without fail since its inception. Before 2012, editing a specific site in a genome meant using zinc finger nucleases: custom-engineered proteins that cost $5,000+ to order, and were never widely adopted because they were so difficult to engineer well. CRISPR replaced this. You order the RNA guide sequence and buy the rest off the shelf, with total cost being as low as $30. This is a strong example of an engineering capability that was once siloed to a handful of specialist labs and is now an undergraduate exercise. Synthesis costs have also plummeted. Synthesizing the poliovirus genome cost about $300,000 in 2002, and the materials would cost under $1,000 today." Fearmongering is bad, but we don't need more AI uplift than we have now for bioweapons to be a major concern. There are active state weapons programs and well-resourced terror groups exploring bioweapons. These groups have already overcome many of the physical barriers, and you have granted that a model could propose a viral genome to synthesize, it could be synthesizable, and it could be infectious. I believe this is cause for alarm and urgent action.
    @EnoReyesBest way to reduce your p(doom) is walk through the causal chain of any AI-driven ex risk scenario. There is a huge amount of hand waving and sci fi reasoning that happens in the middle of these chains. Great thread from @DavidRBellamy
    @Tim_DettmersThis is exactly what I hear again and again when talking to experts. Biorisk is fake. The problem is not designing a dangerous pathogen -- AI is good at that -- it is creating it physically to make it dangerous. And that is what AI cannot do and is easy to safeguard against
    @emollickRT @jrkelly: I run one of these API-driven automated biolabs at @ginkgo and @DavidRBellamy is right -- we are miles away from AI killing us…