The study extends scaling laws to internal model components.
Kosta Derpanis (sabbatical in Zurich)#932
Scaling laws describe how loss changes with scale. Do neurons inside models change predictably too?
We study vision and language models up to 30B params and find systematic scaling in neuron universality, specialization, and selectivity.
Paper+code: https://avdravid.github.io/rosetta-neuron-scaling/
1/n
There is still a lot to learn about how the microscopic organization of large AI models - at the level of individual neurons - changes quantitatively with scale and across diverse training settings. @_AmilDravid uncovers new features of universal neurons connecting directly to semantic structure and scaling. I'm excited for the directions emerging from Amil's work!
Scaling laws describe how loss changes with scale. Do neurons inside models change predictably too?
We study vision and language models up to 30B params and find systematic scaling in neuron universality, specialization, and selectivity.
Paper+code: https://avdravid.github.io/rosetta-neuron-scaling/
1/n
I am very excited about the research directions @_AmilDravid's paper unlocks! I think that there will be a lot of implications here for interpretability, data attribution, and data filtering.
Scaling laws describe how loss changes with scale. Do neurons inside models change predictably too?
We study vision and language models up to 30B params and find systematic scaling in neuron universality, specialization, and selectivity.
Paper+code: https://avdravid.github.io/rosetta-neuron-scaling/
1/n