Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.
The previous observations led to the SymExpLin (SEL) transform. The SymExp part turns additive updates into multiplicative ones, plus learnable curvature params. Then, a standard linear path, which may aid optimization. The network can learn weights to blend each path. https://x.com/torchcompiled/status/2077072941704221005/photo/1
Lastly, my silly self modified the function pretty late on after all the experiments were done and compute was no longer available, but directly learning beta in log space and using this same learned value as the normalized of linear and exp path performs better https://x.com/torchcompiled/status/2077072964319908041/photo/1
We also plot the learned functions as scales shift around by layer. In particular, O_w in attention tends towards softer curvature/more linear on average, MLP down has the most extreme learned curvature. Also it’s commonly seen exp path grows as linear shrinks https://x.com/torchcompiled/status/2077072960318554614/photo/1
At the end we look at the resulting weight distribution and find it’s particularly more heavy tailed than baseline. Even though loss improves, it’s worth taking with a grain of salt if this could mean other side effects, left for future work. https://x.com/torchcompiled/status/2077072956107403424/photo/1
The previous observations led to the SymExpLin (SEL) transform. The SymExp part turns additive updates into multiplicative ones, plus learnable curvature params. Then, a standard linear path, which may aid optimization. The network can learn weights to blend each path. https://x.com/torchcompiled/status/2077072941704221005/photo/1
Lastly, my silly self modified the function pretty late on after all the experiments were done and compute was no longer available, but directly learning beta in log space and using this same learned value as the normalized of linear and exp path performs better https://x.com/torchcompiled/status/2077072964319908041/photo/1
We also plot the learned functions as scales shift around by layer. In particular, O_w in attention tends towards softer curvature/more linear on average, MLP down has the most extreme learned curvature. Also it’s commonly seen exp path grows as linear shrinks https://x.com/torchcompiled/status/2077072960318554614/photo/1
At the end we look at the resulting weight distribution and find it’s particularly more heavy tailed than baseline. Even though loss improves, it’s worth taking with a grain of salt if this could mean other side effects, left for future work. https://x.com/torchcompiled/status/2077072956107403424/photo/1
There’s a few typos and bits that need to be fixed, which will resolve once arxiv updates with the v2 version, and will have code up soon, paper here! https://arxiv.org/pdf/2607.09967 https://x.com/torchcompiled/status/2077072968992378889/photo/1
When reparameterizing our weights like this, we see significant speedups in training up to 1.42x wall clock speedup https://x.com/torchcompiled/status/2077072946317988152/photo/1
optimizer gang may appreciate this @HessianFree @evaninwords @kellerjordan0 @rami_mmo @_arohan_
Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.
When reparameterizing our weights like this, we see significant speedups in training up to 1.42x wall clock speedup https://x.com/torchcompiled/status/2077072946317988152/photo/1