A claimed improvement to transformer initialization for ViTs and language models

A repost shares a claim of a better generic initialization for transformers, including vision transformers (ViTs) and language models (LMs). The original post says it was obtained by training a small looped model.