Reactions from ranked influencers
9 posts
The release of Kimi K3 would be a good time to release GPT-OSS-2, although I don't believe it will happen. The real Open AI currently comes from China, and that won't change for the time being. That should at least give us something to think about.
Kimi K3 2.8T is so large that it will not fit on a single NVIDIA DGX B200, even at FP4. A GB300 NVL72, B300, or MI355X system is required, as each GPU has 288 GB of memory. One optimization that could make Kimi K3 fit on B200 is to gang multiple nodes together and use a technique called WideEP. The issue is that B200 has only 400 Gbit/s of bandwidth between nodes, whereas NVL72 has 18× higher inter-node bandwidth.
Kimi K3 2.8T is so large that it will not fit on a single NVIDIA DGX B200, even at FP4. A GB300 NVL72, B300, or MI355X system is required, as each GPU has 288 GB of memory. One optimization that could make Kimi K3 fit on B200 is to gang multiple nodes together and use a technique called WideEP. The issue is that B200 has only 400 Gbit/s of bandwidth between nodes, whereas NVL72 has 18× higher inter-node bandwidth.
Secondly, although Kimi Delta Attention has up to 10× lower networking requirements for KV-cache transfers, its large weights require even more network bandwidth to implement an optimization called WideEP, which spreads the weights across different GPUs. 3/8🧵
Bring out the B300s.
Kimi K3 2.8T is so large that it will not fit on a single NVIDIA DGX B200, even at FP4. A GB300 NVL72, B300, or MI355X system is required, as each GPU has 288 GB of memory. One optimization that could make Kimi K3 fit on B200 is to gang multiple nodes together and use a technique called WideEP. The issue is that B200 has only 400 Gbit/s of bandwidth between nodes, whereas NVL72 has 18× higher inter-node bandwidth.
Kimi themselves have stated that optimal K3 inferencing will require a rack witha n large scale up domain with at least 64 chips. 7/8🧵
Lastly, Jevons’ Paradox means that making attention more efficient will drive wider AI adoption, which will ultimately require more GPUs, HBM, DRAM, and networking—not less. 8/8🧵