Report
A claimed improvement to transformer initialization for ViTs and language models
The original post says the initialization was obtained by training a small looped model.
TLDR
A repost shares a claim of a better generic initialization for transformers, including vision transformers (ViTs) and language models (LMs). The original post says it was obtained by training a small looped model.
Combined views
—
2 Sources, first seen ago
— likes— comments— saves— reposts
