Announcement
Softmax and cross-entropy implementations reportedly contain two subtle bugs
On X, @jon_barron claims a simple rewrite fixes both bugs and eliminates slingshots during grokking.
TLDR
In an Oct. 23 X post, @jon_barron claimed two bugs affect all existing softmax and cross-entropy implementations, including JAX and PyTorch, and said a simple rewrite eliminates slingshots during grokking. @AlexRoseJo separately alleged a related bug in FlashAttention’s backward pass and claimed CroissAttn fixes it and enables end-to-end FP4 training. The posts do not explain the fixes or demonstrate their effects.
Combined views
63K
4 Sources, first seen ago
