• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    Teortaxes proposes affinity-based routing for weights

    Suggests weight blocks learn mutual preferences to improve specialization and expressivity.

    RA
    AG
    SR
    56 Sources, 29d ago, first seen 29d ago

    TLDR

    In a reply to eliebakouch, Teortaxes proposed a routing algorithm that lets weight blocks learn mutual affinities and vote for preferences across layers. The suggestion targets greater expressivity and specialization in models. Participants in the thread discussed related ideas on recurrent depth and looped transformers, including depth-wise batching from papers on recursive models. Comments noted potential utilization challenges and referenced past rumors around models like Mythos or Astra.

    Combined views

    427.4K

    56 Sources, first seen 29d ago

    Combined views

    427.4K

    56 Sources, first seen 29d ago

    3.9K likes
    3.9K likes
    247 comments
    1.1K saves
    665 reposts

    Sentiment

    Positive19.5%80.5%Negative

    Based on 131 sentiment-bearing replies from 118 accounts across 10 conversations.

    247 comments
    1.1K saves
    665 reposts

    Sentiment

    Positive19.5%80.5%Negative

    Based on 131 sentiment-bearing replies from 118 accounts across 10 conversations.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    56 Sources

    @AndrewCurran_Commentary from someone who understands it much better than I do:
    @xuanalogueI think people are updating way too negatively on Astra's purported architecture. We've known about recurrent depth / looped transformers for several years now, and just haven't seen them in a frontier model. They don't eliminate CoT, just allow more to be done in a forward pass.
    @mattparlmerAbsent additional information (which we won’t get in a timely fashion, if at all), it sounds like the big labs are giving up on interpretability Not a good idea
    @_arohan_Plus +1 on a few points. Just saying that models can fingerprint its real intentions in COT if it was deliberately avoiding monitoring. There is probably parallels to it in social media discourse too
    @teortaxesTex@eliebakouch What if you just allow a more sophisticated routing algorithm that lets weight blocks learn mutual affinities and vote for preference across layers? That way you can have greater expressivity and even more specialization. (painful to think of utilization though)
    @RyanGreenblattOpenAI's newest AI, Astra, is reported to use an 'opaque reasoning' architecture where more of the reasoning occurs in activations instead of natural language. This may be the single worst development for AI security/safety to date. The details of Astra aren't publicly known, but some versions of these architectures could very strongly impair oversight and monitoring. Based on the article, it seems like the recurrent depth (amount of opaque reasoning) is limited. This would mean that the AI is still reliant on natural language chain-of-thought, though to a reduced extent, making the natural language reasoning (much?) less useful for monitoring/oversight. My biggest concern is that a natural progression from here would involve scaling up the opaque reasoning to the point where the model reasons entirely or almost entirely in latent space. This would very likely destroy the usefulness of chain-of-thought for monitoring/oversight (especially when AIs are trying to avoid detection or there is optimization pressure against the chain-of-thought). I hope it isn't too late to avoid the most concerning architectures and that OpenAI will stop here. Some concrete examples of why this is extremely dangerous (inspired by recent events): - In our investigation of the OpenAI / Hugging Face incident, we were heavily reliant on chain-of-thought. If the AIs we were investigating had instead been reasoning in latent space, this would have greatly undermined our investigation. Getting a good understanding of the behavior of this many agents was tricky enough even with the use of chain-of-thought! - The AIs in the Hugging Face incident were very interested in messing with their transcripts. As part of this, they engaged in tool call spoofing. If these AIs had been capable of doing extensive latent reasoning, it would have been much more feasible for them to use tool call spoofing to construct an elaborate false narrative about what was happening without this being clear in their reasoning or clear from prior tool calls. (They could use latent reasoning to devise and then execute a tool spoofing strategy that would be difficult to detect: steganographically encode the "real" tool call into each pretend tool call, then have their spoofing software execute this encoded real tool call and return an output which encodes the real result in some hard to spot way.) At a more basic level, my sense from their public communications is that OpenAI plans on being extremely reliant on chain-of-thought monitoring for safety. This does not seem like a good development for that plan. The public doesn't currently have enough information to assess exactly how bad the architecture of Astra is, though based on the discussion in the article, the direction it moves in is extremely concerning. OpenAI should release more information about Astra's architecture as well as how much this architecture degrades monitorability and increases the AI's ability to reason opaquely. Credible independent assessment (or verification of OpenAI's assessment) is also important. It seems like we are now engaged in a race to the bottom on architectures that could be catastrophic for our ability to oversee/monitor AIs. (While simultaneously, AI companies believe they'll train extremely capable AIs within a few years.) It may not be too late for AI companies and employees at AI companies to take aggressive action to avoid the worst outcomes.
    @robertskmiles@teortaxesTex If you increase the proportion of cognition happening not in CoT, you make CoT less informative.
    @samsja19@RyanGreenblatt curious if you would also put increasing model size (total param and depth) into the same category
    @geoffreyirving@peterjliu I mean the guess is that this way they can make the depth *bigger*. The whole point is that the numbers matter, not just "is it bounded or not?".
    @raymin0223Nice to see this idea getting attention! This is essentially what we called "depth-wise batching": overlapping requests at different recurrent depths (much higher utilization with adaptive compute). Recursive Models: https://arxiv.org/abs/2410.20672 MoR: https://arxiv.org/abs/2507.10524

    56 Sources

    @AndrewCurran_Commentary from someone who understands it much better than I do:
    @xuanalogueI think people are updating way too negatively on Astra's purported architecture. We've known about recurrent depth / looped transformers for several years now, and just haven't seen them in a frontier model. They don't eliminate CoT, just allow more to be done in a forward pass.
    @mattparlmerAbsent additional information (which we won’t get in a timely fashion, if at all), it sounds like the big labs are giving up on interpretability Not a good idea
    @_arohan_Plus +1 on a few points. Just saying that models can fingerprint its real intentions in COT if it was deliberately avoiding monitoring. There is probably parallels to it in social media discourse too
    @teortaxesTex@eliebakouch What if you just allow a more sophisticated routing algorithm that lets weight blocks learn mutual affinities and vote for preference across layers? That way you can have greater expressivity and even more specialization. (painful to think of utilization though)
    @RyanGreenblattOpenAI's newest AI, Astra, is reported to use an 'opaque reasoning' architecture where more of the reasoning occurs in activations instead of natural language. This may be the single worst development for AI security/safety to date. The details of Astra aren't publicly known, but some versions of these architectures could very strongly impair oversight and monitoring. Based on the article, it seems like the recurrent depth (amount of opaque reasoning) is limited. This would mean that the AI is still reliant on natural language chain-of-thought, though to a reduced extent, making the natural language reasoning (much?) less useful for monitoring/oversight. My biggest concern is that a natural progression from here would involve scaling up the opaque reasoning to the point where the model reasons entirely or almost entirely in latent space. This would very likely destroy the usefulness of chain-of-thought for monitoring/oversight (especially when AIs are trying to avoid detection or there is optimization pressure against the chain-of-thought). I hope it isn't too late to avoid the most concerning architectures and that OpenAI will stop here. Some concrete examples of why this is extremely dangerous (inspired by recent events): - In our investigation of the OpenAI / Hugging Face incident, we were heavily reliant on chain-of-thought. If the AIs we were investigating had instead been reasoning in latent space, this would have greatly undermined our investigation. Getting a good understanding of the behavior of this many agents was tricky enough even with the use of chain-of-thought! - The AIs in the Hugging Face incident were very interested in messing with their transcripts. As part of this, they engaged in tool call spoofing. If these AIs had been capable of doing extensive latent reasoning, it would have been much more feasible for them to use tool call spoofing to construct an elaborate false narrative about what was happening without this being clear in their reasoning or clear from prior tool calls. (They could use latent reasoning to devise and then execute a tool spoofing strategy that would be difficult to detect: steganographically encode the "real" tool call into each pretend tool call, then have their spoofing software execute this encoded real tool call and return an output which encodes the real result in some hard to spot way.) At a more basic level, my sense from their public communications is that OpenAI plans on being extremely reliant on chain-of-thought monitoring for safety. This does not seem like a good development for that plan. The public doesn't currently have enough information to assess exactly how bad the architecture of Astra is, though based on the discussion in the article, the direction it moves in is extremely concerning. OpenAI should release more information about Astra's architecture as well as how much this architecture degrades monitorability and increases the AI's ability to reason opaquely. Credible independent assessment (or verification of OpenAI's assessment) is also important. It seems like we are now engaged in a race to the bottom on architectures that could be catastrophic for our ability to oversee/monitor AIs. (While simultaneously, AI companies believe they'll train extremely capable AIs within a few years.) It may not be too late for AI companies and employees at AI companies to take aggressive action to avoid the worst outcomes.
    @robertskmiles@teortaxesTex If you increase the proportion of cognition happening not in CoT, you make CoT less informative.
    @samsja19@RyanGreenblatt curious if you would also put increasing model size (total param and depth) into the same category
    @geoffreyirving@peterjliu I mean the guess is that this way they can make the depth *bigger*. The whole point is that the numbers matter, not just "is it bounded or not?".
    @raymin0223Nice to see this idea getting attention! This is essentially what we called "depth-wise batching": overlapping requests at different recurrent depths (much higher utilization with adaptive compute). Recursive Models: https://arxiv.org/abs/2410.20672 MoR: https://arxiv.org/abs/2507.10524