SonglinYang4 says the team landed NVFP4 in pretraining under the code name “yolo bundle.”
cHHillee points to a blog post explaining how NVFP4 pretraining can work at large scale, saying @jackcookjack initially doubted it would. The codename was less cautious: SonglinYang4 says NVFP4 landed in pretraining as “yolo bundle.”
—
4 Sources, first seen ago
Counts from X are delayed. They’ll show here once X sends them.
Counts from X are delayed. They’ll show here once X sends them.
@cHHilleeWhen I first talked to @jackcookjack, he thought that although he was having fun working on nvfp4 pretraining in academia, he was skeptical that it would ever actually work for actual large-scale runs. As it turns out, he was wrong :^) This blog post explains how it can be done
@SonglinYang4and fun fact: we landed nvfp4 in pretraining under a code name "yolo bundle"
@clarejtbirchOne way to think about hardware is that it presents a set of opinions on arithmetic. One of Blackwell's more optimistic opinions has been that four bits are enough. @jackcookjack and team figured out how to make it work for pretraining all the way through RL:@cHHilleeWhen I first talked to @jackcookjack, he thought that although he was having fun working on nvfp4 pretraining in academia, he was skeptical that it would ever actually work for actual large-scale runs. As it turns out, he was wrong :^) This blog post explains how it can be done@SonglinYang4and fun fact: we landed nvfp4 in pretraining under a code name "yolo bundle"@clarejtbirchOne way to think about hardware is that it presents a set of opinions on arithmetic. One of Blackwell's more optimistic opinions has been that four bits are enough. @jackcookjack and team figured out how to make it work for pretraining all the way through RL:@cHHilleeWhen I first talked to @jackcookjack, he thought that although he was having fun working on nvfp4 pretraining in academia, he was skeptical that it would ever actually work for actual large-scale runs. As it turns out, he was wrong :^) This blog post explains how it can be done@SonglinYang4and fun fact: we landed nvfp4 in pretraining under a code name "yolo bundle"@clarejtbirchOne way to think about hardware is that it presents a set of opinions on arithmetic. One of Blackwell's more optimistic opinions has been that four bits are enough. @jackcookjack and team figured out how to make it work for pretraining all the way through RL:@cHHilleeWhen I first talked to @jackcookjack, he thought that although he was having fun working on nvfp4 pretraining in academia, he was skeptical that it would ever actually work for actual large-scale runs. As it turns out, he was wrong :^) This blog post explains how it can be done@SonglinYang4and fun fact: we landed nvfp4 in pretraining under a code name "yolo bundle"@clarejtbirchOne way to think about hardware is that it presents a set of opinions on arithmetic. One of Blackwell's more optimistic opinions has been that four bits are enough. @jackcookjack and team figured out how to make it work for pretraining all the way through RL:#4
#4
Comments