MiniMax details its M3 sparse attention architecture, claiming a 15.6x decoding speedup at 1 million tokens

Interesting, so M3 will have a yet another sparse attention design, this time with blocks. Looks like a streamlined, simplified NSA. It's cool that we have all labs mapping out the design space.

Reminder that Minimax M2 was supposed to be "Mini", it just turned out to be powerful enough for a whole generation of products. Full-size M3 is likely going to be a big jump.

A bit extreme Even if M3 is a resounding technical and product success, other labs are cooking too. Kimi will come out with K3, for starters, GLM is not going anywhere, Stepfun… it's a question who gets to buy whom.

new minimax sparse attention compared to deepseek v3.2 (DSA) and v4 (CSA)

main changes: - based on GQA not MLA - block level selection like in CSA but attention is done on the real KV, not in the compressed dimension

I just hope it's at least 500B

there's something in the corner of the image mmmmm

MiniMax just teased their Sparse Attention architecture for M3. The benchmarks show 9.7x prefilling speedup and 15.6x decoding speedup at 1M tokens vs M2.

MiniMax deliberately went back to full attention for M2 because efficient attention wasn't production-ready. Their pretrain lead wrote a whole blog post about it in March. Now they're showing a new two-stage approach, lightweight index branch for block selection, then sparse attention only on relevant KV blocks.

Really interesting. And tbh I'm always happy when open source receives new wins.