Report
Nvidia reportedly reaches 423 tokens per second per user on Kimi K3
SemiAnalysis says an upstream vLLM optimization made the 2.8-trillion-parameter model 1.68 times faster on the same hardware.
TLDR
SemiAnalysis reports Nvidia reached 423 tokens per second per user on the 2.8-trillion-parameter Kimi K3 model through a new upstream vLLM optimization, running 1.68 times faster on the same hardware. It credits an Nvidia CentML engineer with writing decode-specialized kernels in Cute DSL using a 12,000-line megakernel, and says Nvidia is working on DeepSeek v4.1 and other models for vLLM.
Combined views
—
3 Sources, first seen ago
