Announcement
Triadic linear attention proposes a three-dimensional state
A user proposes using the outer product of three vectors to move from linear attention’s matrix state to a three-dimensional one.
TLDR
A user introduces triadic linear attention as a proposal to maintain a three-dimensional state using the outer product of three vectors. They argue that linear attention, though arguably a naive RNN, massively outperforms traditional RNNs by maintaining a matrix state.
Combined views
73K
8 Sources, first seen ago
