Reactions from ranked influencers
2 posts
genuine LLM architecture question: why does there have to be multiple sequential MLP layers? why not have many experts all communicating and routing all-to-all? the human brain manages this fine, is multiple layers just a hack to simplify combinatorial complexity for backprop?
@menhguin how are you going to do serial computation that is not written out in cot? the human brain is recurrent over time, layers are an analogue to that since the transformer is not truly recurrent over time steps