Will Depue shows neural network weights retain initialization patterns after training
A series of X posts argues that simple image patterns can stay visible in trained model weights, even after the model learns the task.
In a post on X, Will Depue showed an MNIST classifier initialized with a smiley face and with a photo of his face, then said the pattern was still visible after training. In follow-up posts with optimizer comparisons and a close-up of the face-based run, Depue argued the effect held across learning rates, weight decay and multiple optimizers, while the inherited summary notes that only AdamW with high weight decay erased the watermark.
Combined views
256.5K
13 posts, first seen 17h ago