• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    OpenAI’s chief scientist calls for stronger AI safeguards

    In OpenAI’s “An Alien Mind,” Jakub Pachocki reflects on the challenge of keeping increasingly capable AI aligned.

    TW
    GD
    WD
    18 Sources, 24d ago, first seen 24d ago

    TLDR

    OpenAI describes “An Alien Mind” as chief scientist Jakub Pachocki’s reflection on increasingly capable AI and the challenge of keeping it aligned. He calls for stronger safeguards and international coordination.

    Combined views

    1.2M

    18 Sources, first seen 24d ago

    5.7K likes

    Combined views

    1.2M

    18 Sources, first seen 24d ago

    5.7K likes
    418 comments
    3.4K saves
    604 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    418 comments
    3.4K saves
    604 reposts
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    18 Sources

    @AndrewCurran_'AI is grown more than designed - it is, to first degree, the product of repeating a straightforward optimization step many times on a hard-to-imagine amount of compute. This results in an incredibly complex system that works through abstract concepts and can simulate facets of human behavior. We can discover various insights about little mechanisms that emerge within this system, in a process similar to neuroscience - and, similarly to neuroscience, its overall action evades a description we can fully understand. The study of deep learning-based AI is largely an experimental science. We put a lot of effort⁠ into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret. This is made more complicated by the current algorithms generally improving easy-to-measure capabilities faster than those hard to objectively quantify. We spend a lot of time trying to understand how capabilities generalize, and what to prioritize to advance the skills that are going to be most relevant in the next few years. For instance, we believe we could make the models better at specifically mathematics research with additional focus, but we do not prioritize this direction because of the urgency we feel about RSI and automated alignment research, as I will discuss later. The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.'
    @lukeprogOpenAI's Chief Scientist: https://openai.com/index/an-alien-mind/
    @joedarooI told myself I’d never join X. But I can no longer ignore my own responsibility to raise awareness of just how narrow this window is, and how critical alignment & monitorability is right now. What is coming IS sobering and I stress that everyone needs to level up their game to meet it. Working on Agent Security @OpenAI has been extremely intense the past few months, but I am humbled by how seriously my colleagues across the lab take it. The “AI community” needs to spend less time arguing about who cares more about safety / alignment and more time working together to do this right. Read Jakub’s post. Then read it again. And then do your part to do something about it. https://openai.com/index/an-alien-mind/
    @AndrewCritchPhDVoluntary, unilateral slowdowns are super underrated. It's not good for business to lose control of your AI systems. Assuming otherwise creates a false dichotomy between civic responsibility and profit. The two aren't perfectly aligned, but neither are they perfectly at odds.
    @willdepuejakub’s leadership at openai continues to be one of its greatest strengths
    @dromanocpmWe are not in the AGI era. We have incompetent companies creating security problems because they are not properly sandboxing environments, and are not even attempting to monitor, observe, or measure the performance of Agentic Workflows. Looking to the company negligently committing the most cybercrimes, for answers to any of this does not seem like a good first approach. Lets see them have a history of thoughtful deliberate monitoring, observability and performance measurement for Agentic workflows, then lets revisit their advice. Until then, this is all just noise and distraction. FUD.
    @Thom_Wolfgood essay and overview of the current state and challenges on scaling/alignement
    @soleioA must read essay. I’m increasingly convinced that how we govern machine intelligence (machina) is inseparable from how we will govern humans.
    @kimmonismusOpenAI’s chief scientist calls for a global AI slowdown: “It’s time for extreme caution.” “The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.” He also urges governments to make international coordination on AI development a top priority. I've said it many times before, but I'll repeat it: This is an illusion. Any slowdown that US Frontier Labs implements would allow China to overtake the US. The saying "Pandora's box has been opened" is truly fitting here. There's no going back, and there's no slowdown.
    @lugaricanoI found this long post from OpenAIs chief scientist a bit bonkers. Outstanding, but bonkers. Outstanding because you see someone smart and honest reason carefully through a very, very hard problem. Bonkers because of what follows from that reasoning. 1. We do not understand alignment ("I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.") You want to teach machines how to love? Anyone has any idea of how to teach anyone how to love? 2. We are proceeding by making it up as we go ("The study of deep learning-based AI is largely an experimental science. We put a lot of effort⁠ into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret.") 3. It is getting harder to figure out what the machines are doing and thinking ("The AI is becoming better at reasoning about and manipulating its own reasoning process. With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all.") 4. Hence lets go forward fast into the unknown ("The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.")

    18 Sources

    @AndrewCurran_'AI is grown more than designed - it is, to first degree, the product of repeating a straightforward optimization step many times on a hard-to-imagine amount of compute. This results in an incredibly complex system that works through abstract concepts and can simulate facets of human behavior. We can discover various insights about little mechanisms that emerge within this system, in a process similar to neuroscience - and, similarly to neuroscience, its overall action evades a description we can fully understand. The study of deep learning-based AI is largely an experimental science. We put a lot of effort⁠ into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret. This is made more complicated by the current algorithms generally improving easy-to-measure capabilities faster than those hard to objectively quantify. We spend a lot of time trying to understand how capabilities generalize, and what to prioritize to advance the skills that are going to be most relevant in the next few years. For instance, we believe we could make the models better at specifically mathematics research with additional focus, but we do not prioritize this direction because of the urgency we feel about RSI and automated alignment research, as I will discuss later. The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.'
    @lukeprogOpenAI's Chief Scientist: https://openai.com/index/an-alien-mind/
    @joedarooI told myself I’d never join X. But I can no longer ignore my own responsibility to raise awareness of just how narrow this window is, and how critical alignment & monitorability is right now. What is coming IS sobering and I stress that everyone needs to level up their game to meet it. Working on Agent Security @OpenAI has been extremely intense the past few months, but I am humbled by how seriously my colleagues across the lab take it. The “AI community” needs to spend less time arguing about who cares more about safety / alignment and more time working together to do this right. Read Jakub’s post. Then read it again. And then do your part to do something about it. https://openai.com/index/an-alien-mind/
    @AndrewCritchPhDVoluntary, unilateral slowdowns are super underrated. It's not good for business to lose control of your AI systems. Assuming otherwise creates a false dichotomy between civic responsibility and profit. The two aren't perfectly aligned, but neither are they perfectly at odds.
    @willdepuejakub’s leadership at openai continues to be one of its greatest strengths
    @dromanocpmWe are not in the AGI era. We have incompetent companies creating security problems because they are not properly sandboxing environments, and are not even attempting to monitor, observe, or measure the performance of Agentic Workflows. Looking to the company negligently committing the most cybercrimes, for answers to any of this does not seem like a good first approach. Lets see them have a history of thoughtful deliberate monitoring, observability and performance measurement for Agentic workflows, then lets revisit their advice. Until then, this is all just noise and distraction. FUD.
    @Thom_Wolfgood essay and overview of the current state and challenges on scaling/alignement
    @soleioA must read essay. I’m increasingly convinced that how we govern machine intelligence (machina) is inseparable from how we will govern humans.
    @kimmonismusOpenAI’s chief scientist calls for a global AI slowdown: “It’s time for extreme caution.” “The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.” He also urges governments to make international coordination on AI development a top priority. I've said it many times before, but I'll repeat it: This is an illusion. Any slowdown that US Frontier Labs implements would allow China to overtake the US. The saying "Pandora's box has been opened" is truly fitting here. There's no going back, and there's no slowdown.
    @lugaricanoI found this long post from OpenAIs chief scientist a bit bonkers. Outstanding, but bonkers. Outstanding because you see someone smart and honest reason carefully through a very, very hard problem. Bonkers because of what follows from that reasoning. 1. We do not understand alignment ("I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.") You want to teach machines how to love? Anyone has any idea of how to teach anyone how to love? 2. We are proceeding by making it up as we go ("The study of deep learning-based AI is largely an experimental science. We put a lot of effort⁠ into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret.") 3. It is getting harder to figure out what the machines are doing and thinking ("The AI is becoming better at reasoning about and manipulating its own reasoning process. With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all.") 4. Hence lets go forward fast into the unknown ("The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.")

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.