Standard looping is reportedly compute-optimal for multi-epoch training

A project participant reports that even standard looping is compute-optimal for multi-epoch training—training over repeated passes through the data. They say the optimal number of loops increases with the computational budget and that looping has a useful regularizing effect in that setting. They also identify effective depth as a key factor in both single- and multi-epoch training.