Google DeepMind Unveils HOPE Framework to Quantify Neuron Capacity in Deep Networks
Reactions from ranked influencers
2 posts
2/13 If you want to know how important a neuron is, the standard move is to look at the magnitude of its physical weights (like the L2 norm). But this actually can miserably fail. Deep networks have scale symmetries, so raw weight size is often just an optimization artifact.
3/13 Also context and data distribution matter a ton too. Two neurons might have the exact same weight norm, but one could be "dead" if the data distribution only ever pushes it into the negative zone of its ReLU. We really need a better metric.