Pre-training folks 👀

This is a super interesting observation from the MAI technical report:

Randomly initialized attention naturally behaves like uniform averaging (i.e., the attention matrix is approximately rank-1). They suggested a surprisingly simple training trick.

Feeling confused? Good. Keep reading 🧵