A researcher says frameworks for deriving batch-size scaling rules are less general than they claim.
In a post about their work, the author says optimizers are usually benchmarked at one batch size, with scaling rules used to extend results to others. They argue those rules’ underlying frameworks are less general than claimed and that the best optimizer depends on batch size.
14.7K
6 Sources, first seen ago
@SadhikaMalladiOptimizers are usually benchmarked at one batch size, and scaling rules extend them to different batch sizes. We show: (1) frameworks to derive these rules are not as general as they claim to be, and (2) the best optimizer depends on the batch size. https://batch-size-scaling.com
@ChengleiSiRT @SadhikaMalladi: Optimizers are usually benchmarked at one batch size, and scaling rules extend them to different batch sizes. We show:…
@lambdavikingVery excited to read this! at Ai2 we’ve seen CBS can be subtle, and we’ve been begging @SadhikaMalladi to explain the nuances for us@SadhikaMalladiOptimizers are usually benchmarked at one batch size, and scaling rules extend them to different batch sizes. We show: (1) frameworks to derive these rules are not as general as they claim to be, and (2) the best optimizer depends on the batch size. https://batch-size-scaling.com@ChengleiSiRT @SadhikaMalladi: Optimizers are usually benchmarked at one batch size, and scaling rules extend them to different batch sizes. We show:…@lambdavikingVery excited to read this! at Ai2 we’ve seen CBS can be subtle, and we’ve been begging @SadhikaMalladi to explain the nuances for us@SadhikaMalladiOptimizers are usually benchmarked at one batch size, and scaling rules extend them to different batch sizes. We show: (1) frameworks to derive these rules are not as general as they claim to be, and (2) the best optimizer depends on the batch size. https://batch-size-scaling.com@ChengleiSiRT @SadhikaMalladi: Optimizers are usually benchmarked at one batch size, and scaling rules extend them to different batch sizes. We show:…@lambdavikingVery excited to read this! at Ai2 we’ve seen CBS can be subtle, and we’ve been begging @SadhikaMalladi to explain the nuances for us@SadhikaMalladiOptimizers are usually benchmarked at one batch size, and scaling rules extend them to different batch sizes. We show: (1) frameworks to derive these rules are not as general as they claim to be, and (2) the best optimizer depends on the batch size. https://batch-size-scaling.com@ChengleiSiRT @SadhikaMalladi: Optimizers are usually benchmarked at one batch size, and scaling rules extend them to different batch sizes. We show:…@lambdavikingVery excited to read this! at Ai2 we’ve seen CBS can be subtle, and we’ve been begging @SadhikaMalladi to explain the nuances for us—
Not ranked yet
—
Not ranked yet
