Modified model looping may deliver compute-efficiency gains that grow with scale
A paper’s coauthor says modifying recursive depth, or looping, for model growth can improve scaling exponents during pre-training.
TLDR
Scaling laws predict how loss decreases as computation increases. The paper claims architectural changes can alter those scaling exponents during pre-training. A coauthor says modifying recursive depth, or looping, for model growth can improve the exponents—meaning compute-efficiency gains that increase with scale.
Combined views
4
1 Source, first seen 2h ago
