Reactions from ranked influencers
6 posts
While it is true that Kimi K3 uses Kimi Delta Linear Attention (KDA) in 3 out of every 4 layers and that KDA reduces KV-cache transfer bandwidth by up to 10x compared with comparable full global-attention models, the important missing piece is that Kimi K3 requires WideEP to serve. 2/8🧵
Interesting detail that NVIDIA's paper on LatentMoE already used a hypothetical Kimi-K2-LatentMoE as an example
While it is true that Kimi K3 uses Kimi Delta Linear Attention (KDA) in 3 out of every 4 layers and that KDA reduces KV-cache transfer bandwidth by up to 10x compared with comparable full global-attention models, the important missing piece is that Kimi K3 requires WideEP to serve. 2/8🧵
Because Kimi K3 has 2.8 trillion parameters, even at MXFP4, each forward pass will require 1.5 TB of HBM bandwidth. This means that, even with spec decode, serving it profitably at a reasonable level of interactivity requires aggregating many chips together over a high-bandwidth network, such as the GB300 NVL72. 3/8🧵
Furthermore, with optimizations like incremental KV-cache transfers, prefill only needs to transfer the portions cached by the non-decode instance, so KV transfer does not take up much networking bandwidth relative to WideEP, even before KDA. 7/8🧵
KV-cache transfer between prefill and decode happens only once per turn, while WideEP happens 120+ times per output forward pass, with potentially 500+ output tokens per turn. Thus, the relative KV-cache transfer savings from KDA linear attention are dwarfed by the increase in scale-up bandwidth required for Kimi K3’s massive count of 896 experts. 6/8🧵
Similar to DeepSeek in January 2025, Panicans may think that the AI networking switch TAM will massively shrink because Kimi K3 uses KDA Attention, which reduces KV-transfer networking bandwidth by up to 10x. But the opposite is true, as we explain below. 👇️ 1/8🧵