Hashimoto’s Transformer test: Untying weights helps, but dense still wins
Hashimoto reports that unsharing weights mid-training recovered about 61% of the looped-to-dense validation-loss gap.
TLDR
In Hashimoto’s comparison, training six independent Transformer blocks from the start still won. Turning a looped model’s shared block into six independent copies halfway through training recovered about 61% of the validation-loss gap. Final losses were 2.9333 for dense, 2.9805 for untying midway and 3.0538 for staying looped; lower is better. All three used 614.4 million tokens. A useful catch-up, not a shortcut to first place.
Combined views
1.3K
3 Sources, first seen ago

Comments