NVIDIA releases Nemotron 3 Ultra under the OpenMDW 1.1 license, claiming 5x speedups and 30% lower cost

One of my favorite sniff eval prompts: "You are a pirate who can't help but drop sly reminders he went to Harvard. Explain the Singular Value Decomposition." Nemotron 3 Ultra made a pun about the School of Engineering and Applied Sciences -- SEAS. Which is real.

We have another 65 page frontier model report from Nvidia to read @eliebakouch @stochasticchasm and gang

@eliebakouch @stochasticchasm As usual, lots of datasets to dig into, base model, and a large judge model.

NEW: NVIDIA ships 550B MoE open model for long-running agents. Very exciting times to see more open models to support local long-running coding agents.

open weights and open *data* thank you for everything

Nemotron 3 Ultra is NVIDIA's best model yet and comes with a really great tech report. It focuses mainly on the NVFP4 recipe, and there is a ton of detailed work that went into their Multi-teacher On-Policy Distillation (MOPD) pipeline. A thread of my notes.

The model is 550B-A55B with 22/512 experts + 1 shared expert. Much of the architecture comes from Nemotron 3 Super: - Mamba2-Attention hybrids - LatentMoE (down projecting the input to the MoE). They do a 1/4 down projection of 8192->2048. For reference, MAI did 1/2 down projection only. - 2 MTP with tied weights. - Aggressive GQA with only 2 KV heads and 64 Q heads. - 1 large shared expert with hidden size 10240

The base model basically crushes all other base models while admittedly the other base models are a bit old at this point

They share their NVFP4 quantization recipe in table12. Table 13 I think is showing that not only does their NVFP4 recipe have low bits per element (compression) with diminishing returns after, it also fares well on benches post-quant. It looks like long context has the most variance