Will Depue proposes a distance-based training penalty for weight interpretability, which Ashwinee Panda critiques as heavy-handed

random cute idea: make models more visually interpretable by adding a tiny ‘distance‘ penalty which encourages connecting close in row space. if you look at a weight matrix that learns the identity, everything is usually scrambled hmm what other cute penalties are there