BLASST wins Best Paper at MLSys 2026 for a drop-in training-free dynamic sparse attention mechanism that thresholds online softmax statistics to skip negligible blocks in long-context LLM inference

🚀 BLASST just won Best Paper at #MLSys26! In this paper, we introduce a simple, training-free dynamic sparse attention mechanism that uses a single scalar threshold on online softmax statistics to skip negligible attention blocks. Unfortunately I won’t be there in person, but please say hi to my awesome coauthors! 🙌 Paper: https://arxiv.org/abs/2512.12087

This is an elegant paper; hope to try it out soon.

@jiayiy My favorite genre of paper is 'we stopped doing half the math during inference and the model didn't notice'.

Does dynamic block skipping cause warp divergence on the GPU, or does the kernel mask it somehow?

@jiayiy Hey i am not kidding, i have been working on this since 2024, i had this idea actually about online softmax.. here's the repo https://github.com/MagellaX/StreamAttn

@Infopulsed Hey, thanks for sharing! StreamAttn is quite cool, a clean online softmax streaming kernel in Triton. BLASST reuses the running max stats and mainly focusing on sparse attention, we also provided easy-to-use kernels https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog16_Accelerating_Long_Context_Inference_with_Skip_Softmax_Attention.md. Happy to connect

@jiayiy how is this different from skip softmax from nvidia?

@jiayiy 🐐

@jiayiy Hi Jiayi! Which authors will be there? I’d love to connect with them!

@jiayiy Congrats, Jiayi!!

@jiayiy simple and training-free is doing a lot of work in that sentence. dynamic sparse attention usually pays for it at long-context recall

@jiayiy @xiuyu_l I think recompute QK and do the first pass for max finding in 4-bit (no need for exp so 4-bit actually helps). Then once the sparsity map is determined, gather blocks and compute

@jiayiy we sparse these days mmk. keepin it light. just mad ea sparse kernel =p dig it

@jiayiy 🐮

@jiayiy Congrats. Training-free is the part that grabs me. Way easier to try.

@jiayiy AI researchers really won best paper by teaching the model how to ignore unimportant stuff more efficiently 😭

@_fmla_ this is the tech report (paper) for skip softmax :)

@jiayiy This paper is incredible

@jiayiy Dynamic sparsity is notoriously annoying to implement efficiently. I wrote up some notes a while back on memory-efficient attention variants and why the hardware makes this so hard: https://leetllm.com/learn/flashattention-memory-efficient-attention

@jiayiy congratulations!! :)

@jiayiy Yeah sure, we can talk man! Would love to connect