MiniMax details its M3 sparse attention architecture, claiming a 15.6x decoding speedup at 1 million tokens

new minimax sparse attention compared to deepseek v3.2 (DSA) and v4 (CSA)

main changes: - based on GQA not MLA - block level selection like in CSA but attention is done on the real KV, not in the compressed dimension

MiniMax just teased their Sparse Attention architecture for M3. The benchmarks show 9.7x prefilling speedup and 15.6x decoding speedup at 1M tokens vs M2.

MiniMax deliberately went back to full attention for M2 because efficient attention wasn't production-ready. Their pretrain lead wrote a whole blog post about it in March. Now they're showing a new two-stage approach, lightweight index branch for block selection, then sparse attention only on relevant KV blocks.

Really interesting. And tbh I'm always happy when open source receives new wins.

Something BIG is coming

That's the way. DeepSeek v4 is the gold standard with both KV cache compression *and* indexed attention. But while we can likely live with the memory cost, the computational cost of dense attention for local inference is too heavy at ~100k context already.

Interesting, so M3 will have a yet another sparse attention design, this time with blocks. Looks like a streamlined, simplified NSA. It's cool that we have all labs mapping out the design space.

there's something in the corner of the image mmmmm

Reminder that Minimax M2 was supposed to be "Mini", it just turned out to be powerful enough for a whole generation of products. Full-size M3 is likely going to be a big jump.

A bit extreme Even if M3 is a resounding technical and product success, other labs are cooking too. Kimi will come out with K3, for starters, GLM is not going anywhere, Stepfun… it's a question who gets to buy whom.

I just hope it's at least 500B

@Xianbao_QIAN haha, in several days~

@eliebakouch Tech blog coming soon.

And M3 :)

@SkylerMiao7 Let him cook

@SkylerMiao7 @Xianbao_QIAN Very cool!

Will you all be publishing open weights at the same time that the API is released or will it be a staggered release like M2.7?

@eliebakouch btw just dropped a video explaining the entire deepseek v4 architecture with CSA and HCA:

@leeweiserngmai1 for sure

@SkylerMiao7 Great . Now you have to replace M2.x with M3.x in your bio

@SkylerMiao7 this minimax architecture looks wild but is it actually gonna outperform current sota models or is it just more hype

@SkylerMiao7 LFG! 🚀

@SkylerMiao7 Can it finally support images 🙏