Transformer looping's best strategy shifts with resource limits in preprint tests

Looping repeatedly applies the same transformer, or part of it, using the same weights to add depth without increasing the parameter count. The authors studied dense decoder-only transformers trained from scratch, varying which layers they looped, how many loops they applied and how they passed information between loops. They report experiments involving models with up to 1 billion parameters, training on up to 155 billion tokens, up to 1.07 × 10^21 training FLOPs and up to 32 loops. These are separate reported maxima, not a description of a single run.

The budget changes the answer

At a fixed parameter budget, looping the entire transformer generally produced the strongest quality in the authors' experiments. When training FLOPs were constrained, looping a middle block of layers gave a better return. More repetitions did not always help: the authors report that gains eventually saturated, reversed or became unstable. They also found that small, loop-specific low-rank adapters improved quality, with larger gains as more loops were added.

A cost at decoding time

The preprint reports that decoding slows in proportion to the number of loops. Its Parallel Loop Transformers approach passes hidden states between loops across adjacent token positions, allowing loop stages to run concurrently. In the authors' experiments, the original approach did not consistently retain ordinary looping's quality gains. They report that a revision preserving the current-token embedding and separately normalizing it and the previous-token hidden state before combining them recovered strong performance across model scales and training budgets.