Teortaxes proposes affinity-based routing for weights
Reply suggests letting weight blocks learn mutual preferences to boost specialization and expressivity.
Teortaxes replied to eliebakouch suggesting a routing algorithm that lets weight blocks learn mutual affinities and vote preferences across layers. The proposal aims for greater expressivity and specialization. Other visible replies reference depth-wise batching from papers on recursive models, recurrent depth computation, and past rumors around looped transformers in models like Mythos. One user noted trainability issues with high forward pass counts.
Combined views
60.1K
15 posts, first seen 21h ago
Reactions
13 postsT( 
Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTex@eliebakouch What if you just allow a more sophisticated routing algorithm that lets weight blocks learn mutual affinities and vote for preference across layers? That way you can have greater expressivity and even more specialization. (painful to think of utilization though)
- likes: 14
- replies: 1
- bookmarks: 3
- reposts: 0
X( xuan (ɕɥɛn / sh-yen)@xuanalogueI will retract this / update more negatively if it turns out Astra runs something on the order of 100 forward pass equivalents before emitting a single token, but trainability issues make me think this is quite unlikely.
- likes: 67
- replies: 6
- bookmarks: 22
- reposts: 2
X( X( xuan (ɕɥɛn / sh-yen)@xuanalogueIt's also worth reading the thoughts of people who've actually tried training variants of looped / recurrent depth transformers:
- likes: 3
- replies: 1
- bookmarks: 1
- reposts: 0
EL elie@eliebakouchnote that the latest big model by oai with architecture details was already very deep by today's standards, here is the number of layers for a few models Llama-3.1 405B: 126 GPT3: 96 Kimi K3: 93 Qwen 3.8 max: 92 GLM-5.3: 78 DeepSeek-V4-Pro: 61 gpt-oss 120b: 36 gpt-oss 20b: 24
- likes: 135
- replies: 7
- bookmarks: 33
- reposts: 4
EL elie@eliebakouch@Miles_Brundage it's probably quite deep but not as crazy depth as ppl who beleive that it's a looped transformers think it is
- likes: 6
- replies: 0
- bookmarks: 2
- reposts: 0