Source Posts on X
Cluster sourcesSince the release of GLM 5.2, SGLang team have been constantly working on pushing the performance to the limit, building in the open, and sharing it with everyone. Now you can achieve 535 token/s with latest version of SGLang! Those won't happen without great efforts and support from @radixark @NVIDIAAI and of course @zai_org. Huge respect to @jietang and team for bringing such an amazing model to the world!
Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1 In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens. Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations. Special thanks to @NVIDIAAI for the help in day-0 support of GLM-5.2 NVFP4, and to @Zai_org for IndexShare in SGLang!
Combined views
5.7K
4 posts, first seen 5h ago