Report
Researchers report softmax bugs that erase gradients—or invent them
Barron and coauthors propose fixes for two numerical bugs in standard softmax, log-softmax, and cross-entropy implementations.
TLDR
In “A Fix for Softmax and Cross-Entropy,” Jonathan T. Barron, Dan Gnanapragasam, and Michael Tschannen describe two numerical bugs: one rounds a target logit’s gradient to zero; another produces nonzero gradients even when student and teacher logits match. The authors say their fixes prevent accuracy collapse without weight decay and stop student networks drifting during distillation. @fpedregosa suggests checking implementations against arbitrary-precision arithmetic.
Combined views
359
2 Sources, first seen ago
Comments