NVIDIA Reportedly Achieves 1.68x Inference Speedup for Massive Kimi K3 Model via vLLM Optimizations

Fewer launches for each generated token

The implementation groups work that normally runs as separate GPU operations into persistent kernels written with Nvidia's CuTe DSL. For supported decode steps, one full CUDA graph covers the attention calculations, mixture-of-experts routing and computation, output projection and communication between GPUs. Prefill, mixed batches and unsupported shapes continue through vLLM's standard path.

The change does not alter Kimi K3's weights or scheduling. It is disabled by default and must be selected with vLLM's specialized decode configuration. The merged patch added about 12,800 lines across 50 files, rather than one literal 12,000-line kernel.

The developers also compared the optimized and stock paths on AIME25, GPQA Diamond, GSM8K and a long-context retrieval test. They reported results within about one standard error, while noting high variance in part of the long-context evaluation. In practical terms, the work is a targeted serving optimization: it aims to produce tokens faster on specific Nvidia hardware without changing what the model was trained to do.