Report
The best optimizer may depend on batch size
A researcher says frameworks for deriving batch-size scaling rules are less general than they claim.
TLDR
In a post about their work, the author says optimizers are usually benchmarked at one batch size, with scaling rules used to extend results to others. They argue those rules’ underlying frameworks are less general than claimed and that the best optimizer depends on batch size.
Combined views
2.2K
3 Sources, first seen ago
53 likes2 comments32 saves13 reposts
