Layerless Transformers May Reduce Sparse Circuit Disadvantages
Reactions from ranked influencers
2 posts
@nabla_theta given the existing disadvantages of sparse circuits, would this possibly be less disadvantageous in a "layerless transformer" https://openai.com/index/understanding-neural-networks-through-sparse-circuits/
hmmmn maybe we dont need strict layers (this neuron is always at step N in the forward pass), we might just need some kind of constraint that greatly reduces the search space to 1/n while letting backprop work, no ? theoretically if there was a search algorithm that did this without hardcoding that a neuron always has to be a certain step (which is really what layers are), it would serve just as well sparse training costs 10x due to the gradient flow problem already, so if we're gonna have to solve the gradient flow search problem anyway, we might as well unlock layerless while we're at it. wait! and if layer depth can be learned, we can actually reduce compute required! the attribution problem goes from a resource disadvantage (we have to search 10 layers every time) to a resource advantage (if we search, we can sometimes run backprop on fewer than 10 layers) comparative advantage!