MiniMax details its M3 sparse attention architecture, claiming a 15.6x decoding speedup at 1 million tokens
The design also achieves a 9.7x prefill speedup.
Interesting, so M3 will have a yet another sparse attention design, this time with blocks. Looks like a streamlined, simplified NSA. It's cool that we have all labs mapping out the design space.
Skyler Miao@SkylerMiao7Something BIG is coming
#1496Chubby♨️@KIMMONISMUSMiniMax just teased their Sparse Attention architecture for M3. The benchmarks show 9.7x prefilling speedup and 15.6x decoding speedup at 1M tokens vs M2.
MiniMax deliberately went back to full attention for M2 because efficient attention wasn't production-ready. Their pretrain lead wrote a whole blog post about it in March. Now they're showing a new two-stage approach, lightweight index branch for block selection, then sparse attention only on relevant KV blocks.
Really interesting. And tbh I'm always happy when open source receives new wins.