Announcement
Spotlight's claimed linear computational cost and a potentially large constant factor
A user says each access touches nine cells, each a full d_k×d_v matrix per head.
TLDR
A post quotes a claim that Spotlight combines attention's growing memory and capacity with the linear computational cost of fixed-state recurrent models. The poster says the constant factor seems large: each access touches nine cells, with each cell a full d_k×d_v matrix per head.
Combined views
20.2K
9 Sources, first seen ago
