Reactions from ranked influencers
2 posts
When this is the first line in the Exec Summary, you know this could be a rich Substack: “Kimi K3 is not memory-light: its 2.8T-parameter MoE, 16/896 expert activation and recommended 64+ accelerator deployment still favor HBM and scale-up networking. “ https://funda.ai/reports/deep-llm-kimi-k3-s-kv-cache-is-smaller-why-that-may-actually-be--by76ksrk6v7b
Those ~180MB state matrices are fixed in size, so the manipulation/access cost per token is constant and does not scale with context length. In actual agent workloads — especially with long contexts like Kimi K3’s 1M token window — KV cache IO scales linearly with context length. Every new token generated requires reading the full historical KV, so memory bandwidth demand grows linearly as context accumulates. In long-horizon agent tasks, this quickly becomes the dominant bottleneck during decode. No matter how fast the fixed state manipulation needs to be, it doesn’t change the linear scaling reality of KV cache IO. That’s the core issue in long-context inference for agents.
The argument that deepseek cuts KV down more is moot, because every layer has KV and it grows with context. Additionally searching through and fetching KV is the bottleneck of the operation. For Kimi it’s fetching the previous state layer which is a few megabytes at most and manipulating it. If you don’t do that fast the token throughput collapses and it’s a fixed size no matter your context. So again simplistic arguments with no actual reasoning than “oh look deepseek KV is small too!” This is the logic used by a 3rd grader and removes any nuance. Also DeepSeek is getting mogged by Kimi anyway. Let’s see if they go linear hybrid next too.