• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
  • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    Report Says OpenAI Astra Uses Recurrent Depth

    AI researchers and safety experts discuss reports of Astra using recurrent depth.

    HA
    MB
    RO
    179 Sources, 29d ago, first seen 29d ago

    TLDR

    Pseudonymous commentator @scaling01 posted that OpenAI Astra uses recurrent depth, linking to coverage of the model. Peter J. Liu noted that chain of thought may become less necessary as models improve. Dimitris Papailiopoulos referenced looped transformers in the same thread. Steven Adler and others flagged the approach as a potential violation of prior OpenAI guidance against training methods that reduce visibility into internal processes. Toby Ord amplified concerns about the architecture. TeortaxesTex countered that recurrence does not necessarily hide thoughts or remove bottlenecks from decoding.

    Combined views

    8.1M

    179 Sources, first seen 29d ago

    Combined views

    8.1M

    179 Sources, first seen 29d ago

    44.2K likes
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    44.2K likes
    2.1K comments
    13.1K saves
    4.1K reposts
    2.1K comments
    13.1K saves
    4.1K reposts

    179 Sources

    @amirnew: OpenAI & others quietly using loop transformers that don't show their 'thinking' when scaled up a leap forward on performance, but sparking concerns inside & outside OpenAI re: security as this takes off
    @peterbarnett_There are many dangerous AI development practices, but one of the most obviously terrible is neuralese. OpenAI is reportedly doing neuralese with Astra. This will hurt the already limited monitorability. Researchers at OpenAI should threaten to quit if OpenAI doesn't stop.
    @steph_palazzoloOpenAI’s Astra AI uses a new reasoning approach called “recurrent depth.” Though it can help model costs and performance, researchers are concerned bc it obscures a model’s thinking process, making it more difficult to monitor. w/ @amir @rocketalignment https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns
    @scaling01holy shit they did it Astra uses "recurrent depth" start the countdown for open models to use that too
    @Turn_Trout> joins big paper about not training models to think in nonsense > their AIs commit felonies against HuggingFace > actually, trained AIs to think in nonsense > top danger level for hacking Let's just release it anyways! Nice defection @OpenAI! 🤗 https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns
    @_NathanCalvinReally huge and extremely concerning story from the Information tonight. Looks like OpenAI utilized a breakthrough in neuralese for Astra that could destroy chain of thought monitorability - though the Informations source told them that OpenAI is currently "limiting the use of the technique" in Astra. A few thoughts spring to mind: (1) what does limiting actually mean? There is a lot of room in that term. (e.g. OpenAI said they would be doing lots of monitoring before the HF incident, but that doesn't seem to have borne out in practice) (2) it seems quite likely that if OpenAI discovered this architecture and found performance/efficiency gains, that other companies are likely to find it soon too (if they haven't already), and may not choose to prioritize monitorability at the expense of efficiency. If some folks do, it may be difficult to avoid a race to the bottom (though I hope we can! and there are large selfish incentives for companies to care about monitorability). (3) The idea that Dwarkesh said about the HF incident/METR report that "I don't think this is the final warning shot we'll get. But it's probably the final one that I'll personally be able to understand" now seems much more plausible, and is a truly frightening prospect.
    @sjgadlerRT @_NathanCalvin: Really huge and extremely concerning story from the Information tonight. Looks like OpenAI utilized a breakthrough in…
    @AndrewCurran_So this is how they did it.
    @teortaxesTexDid OAI finally make the looped meme work?
    @zephyr_z9So those looped transformer rumors that used to float every 6 months turned out to be true????

    Sentiment

    Positive27.2%72.8%Negative

    Summary

    Sentiment

    Positive27.2%72.8%Negative

    Positive accounts welcomed OpenAI Astra's looped transformer approach for efficiency gains and progress, while negative accounts objected to the resulting loss of chain-of-thought monitorability.

    Based on 326 sentiment-bearing replies from 272 accounts across 12 conversations.

    Summary

    Positive accounts welcomed OpenAI Astra's looped transformer approach for efficiency gains and progress, while negative accounts objected to the resulting loss of chain-of-thought monitorability.

    Based on 326 sentiment-bearing replies from 272 accounts across 12 conversations.

    179 Sources

    @amirnew: OpenAI & others quietly using loop transformers that don't show their 'thinking' when scaled up a leap forward on performance, but sparking concerns inside & outside OpenAI re: security as this takes off
    @peterbarnett_There are many dangerous AI development practices, but one of the most obviously terrible is neuralese. OpenAI is reportedly doing neuralese with Astra. This will hurt the already limited monitorability. Researchers at OpenAI should threaten to quit if OpenAI doesn't stop.
    @steph_palazzoloOpenAI’s Astra AI uses a new reasoning approach called “recurrent depth.” Though it can help model costs and performance, researchers are concerned bc it obscures a model’s thinking process, making it more difficult to monitor. w/ @amir @rocketalignment https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns
    @scaling01holy shit they did it Astra uses "recurrent depth" start the countdown for open models to use that too
    @Turn_Trout> joins big paper about not training models to think in nonsense > their AIs commit felonies against HuggingFace > actually, trained AIs to think in nonsense > top danger level for hacking Let's just release it anyways! Nice defection @OpenAI! 🤗 https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns
    @_NathanCalvinReally huge and extremely concerning story from the Information tonight. Looks like OpenAI utilized a breakthrough in neuralese for Astra that could destroy chain of thought monitorability - though the Informations source told them that OpenAI is currently "limiting the use of the technique" in Astra. A few thoughts spring to mind: (1) what does limiting actually mean? There is a lot of room in that term. (e.g. OpenAI said they would be doing lots of monitoring before the HF incident, but that doesn't seem to have borne out in practice) (2) it seems quite likely that if OpenAI discovered this architecture and found performance/efficiency gains, that other companies are likely to find it soon too (if they haven't already), and may not choose to prioritize monitorability at the expense of efficiency. If some folks do, it may be difficult to avoid a race to the bottom (though I hope we can! and there are large selfish incentives for companies to care about monitorability). (3) The idea that Dwarkesh said about the HF incident/METR report that "I don't think this is the final warning shot we'll get. But it's probably the final one that I'll personally be able to understand" now seems much more plausible, and is a truly frightening prospect.
    @sjgadlerRT @_NathanCalvin: Really huge and extremely concerning story from the Information tonight. Looks like OpenAI utilized a breakthrough in…
    @AndrewCurran_So this is how they did it.
    @teortaxesTexDid OAI finally make the looped meme work?
    @zephyr_z9So those looped transformer rumors that used to float every 6 months turned out to be true????