Reaction
Brandon Amos flags a numerical trap in KL and Jensen-Shannon divergence
Amos’s paper targets rounding and cancellation errors in comparisons of nearly equal probability distributions.
TLDR
When probability distributions nearly match, rounding and cancellation can distort divergence values and gradients, according to Brandon Amos’s arXiv paper. He proposes reformulations for KL and generalized Jensen-Shannon divergence, with example implementations checked using property-based tests. The catch: Amos says the work demonstrates issues in isolation, not large-scale training, and some reformulations need safeguards against overflow away from equality.
Combined views
14.7K
12 Sources, first seen ago

Comments