Loopie Paper Authors Say Billions Spent Validating Model-Loop Transformers Failed On Large MoE
Reactions from ranked influencers
2 posts
…Pretty wild thing to say but I appreciate the response
@teortaxesTex 1/n Hello, thank you for your question. I am one the first author of this paper. We spent billons of USD to validating the classical Universal Transformer, which we refer to as the “model-loop.” However, we found that it does not scale well to MoE models with billions of params
Hello, thank you for your question. I am the first author of this paper. We spent hundres of millions of USD of compute validating the classical Universal Transformer, which we refer to as the “model-loop.” However, we found that it does not scale well to MoE models with billions of parameters. Please read Section 2.6 of our paper carefully. In the later stages of pretraining, our “layer-loop” approach surpassed the conventional model-loop and continued to maintain its advantage, while also being much more infrastructure-friendly. The conventional model-loop not only underperforms the layer-loop in the later stages of pretraining, but also requires more complicated pipeline parallelism when scaling to larger models. And it places parameters and gradients farther apart, which may lead to slower training. Moreover, outperforming non-loop models with the same theoretical FLOPs is inherently difficult, since non-loop models have full freedom in parameterization. Looping is primarily a method for improving parameter efficiency. Therefore, we chose to match the non-loop baseline based on actual training FLOPs, which we believe is a more practical comparison in industrial-scale pretraining.