Commentary on X
@yzhang_cs True. Upon reading paper Kimi linear at the beginning of this year, I made the decision to conduct an in-depth study of Linear att. Sparse attention is quite complex, and just a patch to a full attention. It have broad space to imagine and extension.
@yzhang_cs We can definitely solve 5M, 10M, to even infinite context length:)
The Conversation on X
2 postsK3 has now crossed the 1M context-length barrier, and DeepSeek's sparse attn has done the same. But what architecture will take us to 5M, 10M, or even longer? I'd always argue that fixed-state linear attn, especially GDN/KDA, is highly competitive here. Hybrid designs are scalable and extrapolate well to longer sequences, while NoPE makes position handling remarkably convenient. Maybe one day we'll see hybrid archs, linear / sparse / full all in one, really take off, breaking through that barrier for long-horizon agent tasks.
funny that K3 is great at making 3Blue1Brown videos, and the Quantile Balancing example in the blog took just a few shots to produce. http://kimi.com/blog/kimi-k3 https://twitter.com/Kimi_Moonshot/status/2077830234955816983
K3 has now crossed the 1M context-length barrier, and DeepSeek's sparse attn has done the same. But what architecture will take us to 5M, 10M, or even longer? I'd always argue that fixed-state linear attn, especially GDN/KDA, is highly competitive here. Hybrid designs are scalable and extrapolate well to longer sequences, while NoPE makes position handling remarkably convenient. Maybe one day we'll see hybrid archs, linear / sparse / full all in one, really take off, breaking through that barrier for long-horizon agent tasks.
funny that K3 is great at making 3Blue1Brown videos, and the Quantile Balancing example in the blog took just a few shots to produce. http://kimi.com/blog/kimi-k3 https://twitter.com/Kimi_Moonshot/status/2077830234955816983