Scaling laws can emerge in models as small as 4M parameters with careful tuning

The team behind “Small-Scale Experiments: Are We There Yet?” says it found scaling laws in models with as few as 4 million parameters, provided their training settings were tuned carefully. These laws use small-model results to predict larger models’ pretraining loss. The updated paper recommends more tuning runs for cheaper small models and fewer for larger ones, while cautioning that even well-tuned predictions become less reliable far beyond the tested range.