AIStandard looping is reportedly compute-optimal for multi-epoch training
For multi-epoch training, a project participant says the optimal number of loops increases with the compute budget and looping has a useful regularizing effect.
TLDR
A project participant reports that even standard looping is compute-optimal for multi-epoch training—training over repeated passes through the data. They say the optimal number of loops increases with the computational budget and that looping has a useful regularizing effect in that setting. They also identify effective depth as a key factor in both single- and multi-epoch training.
Combined views
42.6K
6 Sources, first seen 13h ago
Standard looping is reportedly compute-optimal for multi-epoch training
For multi-epoch training, a project participant says the optimal number of loops increases with the compute budget and looping has a useful regularizing effect.