Users praise the NVFP4 RL blog post for its clear visualization of stability tradeoffs and practical results showing reward parity with BF16 plus efficient serving with no degradation.
Based on 5 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@nrehiew_ Reward curve parity between NVFP4 and BF16 is the real result. Memory savings are predictable. Matching baseline reward at 4-bit means the quantization noise isn't corrupting the RL signal.
This is an insanely good visualization. With one look, you instantly understand how NVFP4 works https://x.com/nrehiew_/status/2076654138885358022/photo/1
@nrehiew_ Recipe-style work is what I look for. Staleness thresholds, precision tradeoffs, throughput boundaries. This is the layer that makes RL practical.
The final results are pretty cool. https://x.com/nrehiew_/status/2076654158816649456/photo/1
The difference in precision for forward vs backward led to gradient spikes so the first thing they do is use the dequantized bf16 value for backward. ie Dequant(quant(x)) to bake in the clipping While reward curve grew faster, the KL and grad norm were significantly higher (in NVFP4, MXFP8 was fine). Interestingly enough, when they switched back to adam, the spikes were gone; I assume the KL problem still exists?
@nrehiew_ Reward curve parity between NVFP4 and BF16 is the real result. Memory savings are predictable. Matching baseline reward at 4-bit means the quantization noise isn't corrupting the RL signal.
This is an insanely good visualization. With one look, you instantly understand how NVFP4 works https://x.com/nrehiew_/status/2076654138885358022/photo/1
@nrehiew_ Recipe-style work is what I look for. Staleness thresholds, precision tradeoffs, throughput boundaries. This is the layer that makes RL practical.
The final results are pretty cool. https://x.com/nrehiew_/status/2076654158816649456/photo/1
The difference in precision for forward vs backward led to gradient spikes so the first thing they do is use the dequantized bf16 value for backward. ie Dequant(quant(x)) to bake in the clipping While reward curve grew faster, the KL and grad norm were significantly higher (in NVFP4, MXFP8 was fine). Interestingly enough, when they switched back to adam, the spikes were gone; I assume the KL problem still exists?
The next step is to scale the representation range to be between +- 4 which reduces the largest possible relative error. For both activations and weights, both schmes are computed and the one with the smallest representation error is kept. The problem is that this is pretty expensive and so there is a bunch of kernel tricks needed.
It is worth starting by looking at the pretraining nvfp4 recipe from Nemotron: https://x.com/nrehiew_/status/2062553062984872015?s=20 tldr: - Hadamard transforms - Selective high precision - Stochastic rounding
Some notes on the humans& 4bit RL blog. Big fan of this type of recipe style work https://x.com/nrehiew_/status/2076654135559233857/photo/1
Lastly, following nemotron, the following are kept in bf16: - the last 15% - the shared expert https://x.com/nrehiew_/status/2076654155574465018/photo/1
Users praise the NVFP4 RL blog post for its clear visualization of stability tradeoffs and practical results showing reward parity with BF16 plus efficient serving with no degradation.
Based on 5 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.
The next step is to scale the representation range to be between +- 4 which reduces the largest possible relative error. For both activations and weights, both schmes are computed and the one with the smallest representation error is kept. The problem is that this is pretty expensive and so there is a bunch of kernel tricks needed.
It is worth starting by looking at the pretraining nvfp4 recipe from Nemotron: https://x.com/nrehiew_/status/2062553062984872015?s=20 tldr: - Hadamard transforms - Selective high precision - Stochastic rounding
Some notes on the humans& 4bit RL blog. Big fan of this type of recipe style work https://x.com/nrehiew_/status/2076654135559233857/photo/1
Lastly, following nemotron, the following are kept in bf16: - the last 15% - the shared expert https://x.com/nrehiew_/status/2076654155574465018/photo/1