NVIDIA details Nemotron 3 Ultra 550B pre-training using NVFP4, LatentMoE, and 20 trillion tokens

Now back to the fun part about stability during NVFP4 training. When faced with divergence around 40% of training, they roll back to using FP32 gradient. Previous NVIDIA work which showed that reverting to high precision can kind of recover some loss divergence - provided that its done earlier on in the training as later stages have too low a learning rate to recover.

1) The first divergence happens because of the MTP loss which is scaled to 0.1. The gradient from the MTP heads is lost in BF16. From the figure we see that MTP loss starts diverging before overall training loss. This was resolved by using FP32 gradient recipe

Post-training uses expert RLVR + MOPD merging of the experts. SFT data is a mix of : - long context data - GPTOSS-120B generated synthetic data for STEM/Instruction Following. Interestingly during SFT, they mask out thinking tokens and do not train on traces - Multilingual Safety - An expanded search dataset of 21.7K trajectories with MiniMax 2.5 and GLM5.1 as teachers - Terminal use data where DSV3.2 is used within Terminus 2 Agent - Synthetic Conversational using GLM 5 and Nemotron GenRM as the preference model - SWE trajectories using several harnesses (OpenHands, SWE-Agent, MiniSWEAgent, Opencode). The second image below has some info on the heuristics they used to filter for trajectories - DSV3.2 + Nemotron Cascade/Math for Math data - GPTOSS120B competitive coding problems - CUDA Kernel data using DeepSeek R1(?) and GPT OSS 120B for both pytorch and natural language to kernel problems. They select based on benchmarking and also train for repair with the model even having access to the profiler. (Super hyped for this)

Not much info on the RL data beyond the standard domains. The algorithm seems to be GRPO with IS masking so pretty standard They say they have a "Gaussian" sampling strategy for the curriculum. I'm not entirely sure what this means so leaving this section from Nano here

Next, a massive section on MOPD. Algorithm wise, they actually have 2 MOPD iterations. Loss is just reverse KL without any environment reward loss term. Their version of MOPD has 3 async parts (rollout, teacher and learners). So they compute the log probs from all 3 with the KL being between the teacher and the learner. They do a similar importance ratio masking where the ratio that is masked is between the rollout and teacher while the teacher-learner is clipped. They have a ton of info on the data/process used for each teacher. It super detailed and so i think its pointless for me to distill it here so read that section for this part

Because all the experts are trained individually and differently, they say that MOPD cannot be applied naively. What ends up happening is that the student is too different from the teacher. They do a very light SFT stage on each teacher's data as warmup. The benefit is most pronounced in agentic domains vs reasoning ones. I suspect its probably something to do with longer rollouts/multi turn (?)

They now have a long section on RL infra. Training is 1 step async off policy and aggressively uses 5 MTP spec decoding to speed up rollout Much of the failures they encounter are due to sandbox/toolcalling/generation. Much of their optimizations target reliability/efficiency here. The table below is a nice summary but again I dont see a point in summarizing since the section has so many cool details. Go check it out! This is just a quick list: - Slurm launch overheads - Make EP AllToAll toplogy aware to only be within NVLink domains and not InifiniBand between racks. (20%! throughput improvement) - In a NVL72 rack, gpus are affiliated to nearby NUMA nodes for faster memory access. Without special care, workers on a GPU could be binded to a remote node rather than the local faster memory. (10% throughput) - Async checkpointing - Compilation times taking ages so cache and share compiled artifacts. (I have a tweet about this hahaha - Multinode vllm instablity -All nodes were reading and crashing the shared storage.

For benchmarking, their table has 0 bolding or highlighting which makes it annoying to compare so I got GPT-Image to annotate. Kimi K2.6 is quite insane still but Nemotron 3 ultra fares very well on non agentic tasks

There is also a bunch of miscellanous quantization stuff. 1) Scaling algorithm selection 2) Quantization speed/performance 3) Int8 Stochastic Rounded Mamba cache (important since mamba has larger cache at lower context lengths) Finally, they show minimal degradation post-quant

Lastly, a section on inference. Arch wise, its already pretty inference friendly with latentmoe allowing for more routed experts, hybrid mamba2, mtp etc. First, Ultra has lower sparsity which means more flops at prefill where it trails Qwen with ~2x more activated params. But, it dominates at decode mainly due to the SSM layers Optimal MTP length is 6 for almost 3x faster throughput. For the Mamba state, they snapshot at each step to facilitate rollbacks on rejection.

At small batch sizes, we want to do TP since the allreduce comms amount is lower with smaller activation sizes. At larger batch sizes, we much rather do EP which again is a nice benefit of NVL72 They recommend using FlashInfer's NVLinkOneSided for AlltoAll (AllGather + ReduceScatter duplicates tokens) for a 5% improvement. This is pretty cool. My understanding of NVLinkOneSided is that source device directly writes to the target's symmetric memory. Normally you need to read/postprocess this data before writing back to HBM before the compute. Instead of doing that, we write tensor views (not the data) to the symmetric memory buffer, eliminating the problem where the buffer is too small to receive the data wholesale. The target GPU than reads the data remotely in place. Again, I suppose this is only possible at scale thanks to NVL72.

Loved this tech report (second nice one in a week!). It seems that NVIDIA is really pushing everyone to use NVFP4 at larger scales. As such, really appreciate the transparency in sharing the recipe, including the instabilities (cutting training to 20T tokens). Also thought the MOPD was super detailed and not what I expected from NVIDIA, so pretty excited to see future work in this area.