Teortaxes proposes affinity-based routing for weights

In a reply to eliebakouch, Teortaxes proposed a routing algorithm that lets weight blocks learn mutual affinities and vote for preferences across layers. The suggestion targets greater expressivity and specialization in models. Participants in the thread discussed related ideas on recurrent depth and looped transformers, including depth-wise batching from papers on recursive models. Comments noted potential utilization challenges and referenced past rumors around models like Mythos or Astra.