Critics questioned the JAX benchmark and choice of C.
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
Positive users praise SpaceX's custom C-based AI training stack for 220k GPUs as deep engineering that could deliver major speed gains, while negative users direct personal insults at Elon Musk.
Is this framework open source?
No. SpaceX’s custom C-based AI training stack is described throughout as an in-house framework built for its private 220k-GB300 cluster, with no announcements or indications of open-source release.
Next will be writing the inference stack in C for simultaneous high-speed RL across a large block of GB300s.
(We do use a little C++ tbh, but not much)
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
Yes.
It’s not that we’ve discovered some magic bullet, but rather that JAX, or at least the open source version of it, is mostly optimized for small to medium-sized training runs on Google TPUs, whereas we need to massive training runs on Nvidia GPUs.
Pipeline parallelism is essential and crushes fully-sharded data parallelism at scale.
And C will compile to the most efficient binary short of assembly. Maybe we will do a little assembly too.
There’s a ton of headroom to better optimize AI training and inference.
Note, I am posted this to encourage those who enjoy getting incredible performance out of hardware to join SpaceX
Next will be writing the inference stack in C for simultaneous high-speed RL across a large block of GB300s.
(We do use a little C++ tbh, but not much)
The fact that JAX was even mentioned makes me think of two things: 1) xAI still needs better ML people 2) PyTorch is stagnating and probably is not going to recover :-( It is such an unparalleled achievement, but lost key people, exiled to FAIR now...
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
@KazimAIZJU With great {malloc} power comes great responsibility
There’s a ton of headroom to better optimize AI training and inference.
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
Improving CPU speed by 10x should not affect training speed essentially at all.
The CPU's main job is to kick off the real work on the GPU. If your kernels are sane (fused etc), the time to launch a kernel on the CPU is <<1% of the kernel runtime, even in Python.
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
@elonmusk Elon, trust me, this makes no sense. Someone is strongly overclaiming or overpromising to you.
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
@wholemars @swiftg008 I agree
@elonmusk Truth requires it
j...jax????
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
PufferLib trains small task-specific reinforcement learning models at up to 20M steps/second in 5,000 lines of CUDA C on a single GPU. Stop using awful DSLs just because they are pretending to be Python!
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
"/goal rewrite jax in rust"
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
@elonmusk I hope this helps someone:
@elonmusk @swiftg008 Pure C > C++
Modern C++ is an abomination
You are right that launch latency is negligible for kernels, but that’s not at all what he’s talking about here.
He’s talking about optimizing a training run or inter-node GPU orchestration at a huge 220k multi-node scale. The problem is how do you coordinate and schedule compute between nodes efficiently, and the CPU is the control plane for that. If you just naively use XLA you leave tons of performance on the table because it’s not going to map to your exact topology and network, XLA is better for a single device or single node.
Therefore, it’s no surprise that he specifically calls out NICs, because large scale training workloads are communication bound and the overhead of the communication compounds. PCIe round trips look negligible, but multiply them by millions of ops per step across thousands of NICs all blocking GPUs from doing work and it becomes a substantial bottleneck. This is why everyone is looking at photonics for an easy way out rather than doing what he’s trying to do lol cause it’s hard. Optimizing the network for just the topology gets you most of the way there and makes all to all and all reduce traffic way more efficient.
And at this scale the tail latency starts to matter too. A single slow node can stall the entire cluster, and any jitter or scheduling variance compounds that across thousands of nodes. Which is why using C in the right places is absolutely necessary.
Sounds great. But to be honest, here’s what my non-tech brain just read:
“SpaceX has almost finished something 1.0 something AI ABC. 220K to buy Great Britain, and 300 + 800 nicotine patches for heavy metal singers with pipes in a parallel universe.
And Jax is training & will order a magnum.”
@elonmusk
@elonmusk C is for the Cracked.
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
10x potential speedup implies that their current MFU is under 10% 😂
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
