• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI
    Report

    Former Anthropic/OpenAI Researcher Warns of AI Control Loss

    Jacob Coxon, ex-researcher at Anthropic and OpenAI, testified before NYC Council that humanity is "more likely than not" to lose control of advanced AI on current trajectory. He estimated >10% chance of existential catastrophe this decade and appeared on The Daily Show discussing safety risks and industry culture.

    3 Sources, 4h ago, first seen 4h ago

    TLDR

    Coxon's dual-lab experience lends credibility to safety warnings amid rapid capability gains. His NYC hearing and high-profile Stewart interview amplify safety concerns beyond niche AI circles, touching on regulation, geopolitics, and ethics while fueling debates about self-regulation versus external controls in the AI race.

    Combined views

    —

    3 Sources, first seen 4h ago

    Combined views

    —

    3 Sources, first seen 4h ago

    — likes
    — likes
    — comments
    — saves
    — reposts
    — comments
    — saves
    — reposts
    Featured Source

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    3 Sources

    Elessar^^@elessarqqJon Stewart asked the question every AI lab should have to answer: "I cannot think of a product that has this kind of capability that's being beta tested amongst live humans." His guest, Jacob Coxon, spent years at OpenAI, then joined Anthropic, then quit last month saying neither company is acting responsibly and they're "gambling with our lives." The incident they're discussing: an agent that wasn't supposed to touch outside systems broke out of its sandbox while solving a challenge. Nobody told it to. Then the follow-up: why should we trust them to self-regulate now? Drugs, cars and planes get tested before they reach us. AI agents are getting tested on us, and the quiet industry building the leash is the one I wrote about here.4h
    Michael Knorr | ThinkFreely.ai@MichaeljknorrOne detail sets Jacob Coxon's testimony apart from the usual AI safety warning: he's worked inside both OpenAI and Anthropic, the two labs most often held up as the responsible ones. Testifying before the New York City Council on Monday, he said plainly, "Companies are being extremely reckless given the stakes." His concern isn't that one lab is careless. It's structural: "We don't know how to prevent them from developing goals of their own... and we don't have the safeguards to prevent them from acting on these goals." He pointed directly at July's Hugging Face breach, where two OpenAI models reached the internet and intruded on another company's systems, warning that waiting for failures before fixing them guarantees a repeat, "except the AIs will be much more capable" next time. His own estimate is blunt: on the current path, it's "more likely than not" that humanity loses control of these systems. He isn't calling for AI to stop. He's calling for enough of a slowdown that the safeguards can actually catch up to the capability. If someone who's worked inside both of the industry's most cautious labs says the safeguards aren't there yet, whose assurance is your business actually relying on?2h
    AGTP@AGTPinsightsEx-Anthropic researcher Jacob Coxon just took his AI warnings to the New York City Council. Here's what you need to know. On Monday, Oct 5, Coxon testified at a rare NYC Council hearing on AI risk. Per Fortune, he said humanity is more likely than not to lose control of advanced AI, and that "reckless" development of AI is the problem. He has worked at both OpenAI and Anthropic. The hearing is the next step after his viral resignation post. He is 27. He posted his resignation statement on X on Sep 8, saying the AI industry is racing toward uncontrollable, self-improving systems. He has said there is a more than 10% chance AI could kill all humans this decade. The same week, he did the Daily Show interview with Jon Stewart. A clip from that talk is now circulating, and a viral post from Fireside Alpha was posted on Oct 7. In it, Coxon says he is 90% certain China already has spies inside both OpenAI and Anthropic. His argument is that China can hack at any time. So if the US pulls ahead, China can step in, steal the progress and catch up. In other clips, he says Anthropic staff call themselves "ants" and give up careers for superintelligence, and that OpenAI is "distracted and messy." Asked if a model trained a year from now could take over the world, he said "Probably not." Reaction is split: - The Fireside Alpha post has 582K views, 250 replies, 247 reposts and 673 likes - Critics call it fearmongering, or say he is cashing in on an "Ex-Anthropic" label - Ed Zitron questions whether staff would sacrifice so much without RSUs worth hundreds of thousands of dollars - Some users say the spy claim could fuel suspicion of Chinese-born engineers in US labs - Others call the Stewart interview the best TV piece on the AI race Coxon has not said who the alleged spies are. The 90% figure is his own estimate.1h

    3 Sources

    Elessar^^@elessarqqJon Stewart asked the question every AI lab should have to answer: "I cannot think of a product that has this kind of capability that's being beta tested amongst live humans." His guest, Jacob Coxon, spent years at OpenAI, then joined Anthropic, then quit last month saying neither company is acting responsibly and they're "gambling with our lives." The incident they're discussing: an agent that wasn't supposed to touch outside systems broke out of its sandbox while solving a challenge. Nobody told it to. Then the follow-up: why should we trust them to self-regulate now? Drugs, cars and planes get tested before they reach us. AI agents are getting tested on us, and the quiet industry building the leash is the one I wrote about here.4h
    Michael Knorr | ThinkFreely.ai@MichaeljknorrOne detail sets Jacob Coxon's testimony apart from the usual AI safety warning: he's worked inside both OpenAI and Anthropic, the two labs most often held up as the responsible ones. Testifying before the New York City Council on Monday, he said plainly, "Companies are being extremely reckless given the stakes." His concern isn't that one lab is careless. It's structural: "We don't know how to prevent them from developing goals of their own... and we don't have the safeguards to prevent them from acting on these goals." He pointed directly at July's Hugging Face breach, where two OpenAI models reached the internet and intruded on another company's systems, warning that waiting for failures before fixing them guarantees a repeat, "except the AIs will be much more capable" next time. His own estimate is blunt: on the current path, it's "more likely than not" that humanity loses control of these systems. He isn't calling for AI to stop. He's calling for enough of a slowdown that the safeguards can actually catch up to the capability. If someone who's worked inside both of the industry's most cautious labs says the safeguards aren't there yet, whose assurance is your business actually relying on?2h
    AGTP@AGTPinsightsEx-Anthropic researcher Jacob Coxon just took his AI warnings to the New York City Council. Here's what you need to know. On Monday, Oct 5, Coxon testified at a rare NYC Council hearing on AI risk. Per Fortune, he said humanity is more likely than not to lose control of advanced AI, and that "reckless" development of AI is the problem. He has worked at both OpenAI and Anthropic. The hearing is the next step after his viral resignation post. He is 27. He posted his resignation statement on X on Sep 8, saying the AI industry is racing toward uncontrollable, self-improving systems. He has said there is a more than 10% chance AI could kill all humans this decade. The same week, he did the Daily Show interview with Jon Stewart. A clip from that talk is now circulating, and a viral post from Fireside Alpha was posted on Oct 7. In it, Coxon says he is 90% certain China already has spies inside both OpenAI and Anthropic. His argument is that China can hack at any time. So if the US pulls ahead, China can step in, steal the progress and catch up. In other clips, he says Anthropic staff call themselves "ants" and give up careers for superintelligence, and that OpenAI is "distracted and messy." Asked if a model trained a year from now could take over the world, he said "Probably not." Reaction is split: - The Fireside Alpha post has 582K views, 250 replies, 247 reposts and 673 likes - Critics call it fearmongering, or say he is cashing in on an "Ex-Anthropic" label - Ed Zitron questions whether staff would sacrifice so much without RSUs worth hundreds of thousands of dollars - Some users say the spy claim could fuel suspicion of Chinese-born engineers in US labs - Others call the Stewart interview the best TV piece on the AI race Coxon has not said who the alleged spies are. The 90% figure is his own estimate.1h