Rubin Applies 2:4 Sparsity to Attention Outputs for Bandwidth Savings
Reactions from ranked influencers
2 posts
This sparsity idea is super interesting and out of left field. For every local group of 4 elements, we keep the 2 highest magnitude ones and store their indices - implicitly zeroing the rest Rubin then performs the MMA directly on the compressed tensor + indices which saves bw
4/ Attention Sparsity: Rubin can load the outputs of Q * K in a 2:4 sparse compressed format, decreasing BW and storage requirements!
I'm a little confused by why/how this would work and can only speculate that this is co-designed with the Frontier Labs' architecture in mind. For attention, doing local sparsity makes little sense since tokens in the same neighbourhood will, by construction, have similar attention weights For the MLP, doing sparsity across the reduction dimension can only be designed with some form of QAT in mind
This sparsity idea is super interesting and out of left field. For every local group of 4 elements, we keep the 2 highest magnitude ones and store their indices - implicitly zeroing the rest Rubin then performs the MMA directly on the compressed tensor + indices which saves bw https://twitter.com/KranenKyle/status/2079608224215708002