Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.
I think this is one of the most fascinating projects I’ve tinkered on yet. Parameterizing our weights as a learned weighted blend of symexp and regular linear weights gives up to ~1.42x training speedup wallclock, and can be fused back into standard weights for deployment🧵 https://x.com/torchcompiled/status/2077072920992714790/photo/1
The exploration started with a frustration that preconditioning is clearly beneficial, but it doesn’t really acknowledge the “optimization distance” some parameters have to travel. In pretrained models we observe a heavy tailed distribution of weights. https://x.com/torchcompiled/status/2077072925296042429/photo/1
I think this is one of the most fascinating projects I’ve tinkered on yet. Parameterizing our weights as a learned weighted blend of symexp and regular linear weights gives up to ~1.42x training speedup wallclock, and can be fused back into standard weights for deployment🧵 https://x.com/torchcompiled/status/2077072920992714790/photo/1
The exploration started with a frustration that preconditioning is clearly beneficial, but it doesn’t really acknowledge the “optimization distance” some parameters have to travel. In pretrained models we observe a heavy tailed distribution of weights. https://x.com/torchcompiled/status/2077072925296042429/photo/1
Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.