A proposed explanation for double descent: memorization giving way to reusable patterns

Drawing on mechanistic interpretability blogs, a post describes how models with little training data can treat individual examples as features. As datasets grow, it says models shift toward reusable features shared across examples, undoing earlier memorization. The post links this transition to double descent’s bump in test loss. It says generalization seems to emerge once each underlying feature appears repeatedly in different combinations—roughly 10 occurrences per feature in the simplest experiments. Repeated datapoints, it adds, compete with reusable features for model capacity.