Reduced interference may help larger AI models learn rare tasks

Huang and coauthors tested why larger language models learn tasks smaller ones miss. In synthetic experiments and pretraining runs with OLMo models ranging from 4 million to 4 billion parameters, only the larger models learned infrequent, complex tasks. The researchers attribute that advantage to reduced gradient interference: training on common tasks is less likely to overwrite features built up for rare ones.