DeepMind Researchers Introduce HOPE Neuron Capacity Framework
Framework models neurons as rank-1 operators in Hilbert space for data-free analysis.
Google DeepMind and UC Berkeley researchers introduce HOPE, a Hilbert space-based mathematical framework that models neurons as rank-1 operators. It quantifies neuron capacity rigorously without data, addressing failures of weight magnitude metrics due to scale symmetries and data distribution effects. The method evaluates capacity analytically in closed form using BatchNorm statistics and the Central Limit Theorem. The work aims to better deconstruct what deep networks have learned and support network compression. The paper is available on arXiv.
Combined views
37.9K
7 posts, first seen 21h ago
DeepMind Researchers Introduce HOPE Neuron Capacity Framework
Framework models neurons as rank-1 operators in Hilbert space for data-free analysis.
Google DeepMind and UC Berkeley researchers introduce HOPE, a Hilbert space-based mathematical framework that models neurons as rank-1 operators. It quantifies neuron capacity rigorously without data, addressing failures of weight magnitude metrics due to scale symmetries and data distribution effects. The method evaluates capacity analytically in closed form using BatchNorm statistics and the Central Limit Theorem. The work aims to better deconstruct what deep networks have learned and support network compression. The paper is available on arXiv.
Reactions from ranked influencers
7 posts
Excited to share about what I’ve been working on over the past year: quantifying the capacity of a neuron😅 This led to a mathematical framework we call HOPE, which lets us rigorously deconstruct what deep networks might have learned. Paper: https://arxiv.org/abs/2607.21366
2/13 If you want to know how important a neuron is, the standard move is to look at the magnitude of its physical weights (like the L2 norm). But this actually can miserably fail. Deep networks have scale symmetries, so raw weight size is often just an optimization artifact.
3/13 Also context and data distribution matter a ton too. Two neurons might have the exact same weight norm, but one could be "dead" if the data distribution only ever pushes it into the negative zone of its ReLU. We really need a better metric.
7/13 Usually, you would approximate the integral using finite data by passing massive datasets through the network. But HOPE evaluates it analytically in closed form! Since neurons perceive inputs as 1D Gaussians (by the Central Limit Theorem), we can build an exact continuous surrogate using BatchNorm stats.
@TheGradient holy shit these are insanely well-motivated experiments tackling a lot of longstanding confounders in interpretability illusions. woah.
@TheGradient Some results take a year to mature. Congrats.