Moonshot AI Open-Sources FlashKDA Delta Attention Kernels
Beijing firm releases CUTLASS code for Kimi linear attention system.
Moonshot AI, the Beijing company behind the Kimi chatbot and LLM series, published its FlashKDA repository on GitHub. The library supplies high-performance CUTLASS-based kernels for Kimi Delta Attention, designed as a drop-in backend for flash-linear-attention frameworks. The release coincides with FLA v0.5.2, which incorporates the kernels along with added KDA CP support. Commenters describe the implementation as one of the strongest available for linear attention and note related work on attention residuals that preserve access to prior states. The code targets efficient prefill for frontier-scale models.
We've open-sourced FlashKDA, our high-performance CUTLASS-based implementation of Kimi Delta Attention kernels. It delivers 1.72×–2.22× prefill speedup over the flash-linear-attention baseline on H20, and works as a drop-in backend for flash-linear-attention. Explore on GitHub:
Reactions from ranked influencers
6 postsprobably the best implementation of linear attention to date
We've open-sourced FlashKDA, our high-performance CUTLASS-based implementation of Kimi Delta Attention kernels. It delivers 1.72×–2.22× prefill speedup over the flash-linear-attention baseline on H20, and works as a drop-in backend for flash-linear-attention. Explore on GitHub:
Quick sync alongside the K3 report: FLA v0.5.2 is out 🚢 - KDA fused kernels w/ FlashKDA backend - KDA CP support - Fused attnres Enjoy it! cc @SonglinYang4 @uniartisan
Releasing the model weights and technical report of Kimi K3. Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window. New model architecture: 2.5x the intelligence per unit of compute, not just more params. Alongside…
They developed custom kernels (optimized CUDA code) for their new attention, and published it on GitHub. I just noticed they used Claude Fable to do the latest commit (forgot to scrub from PR description) Cue @DarioAmodei 's wrath 😅
🍿 forget distillation, the real advantage is using Claude Fable to do research
They developed custom kernels (optimized CUDA code) for their new attention, and published it on GitHub. I just noticed they used Claude Fable to do the latest commit (forgot to scrub from PR description) Cue @DarioAmodei 's wrath 😅
Another innovation which I've not seen in frontier models is Attention Residuals. They make an astute observation that what made attention effective in the first place beyond standard RNNs is the ability to access state from previous steps, instead of summarized in the state…
They developed custom kernels (optimized CUDA code) for their new attention, and published it on GitHub. I just noticed they used Claude Fable to do the latest commit (forgot to scrub from PR description) Cue @DarioAmodei 's wrath 😅