Jiaxin Wen, UC Berkeley CS PhD student and part-time Anthropic researcher, argues models acquire little meaningful knowledge early in pre-training and says this invalidates strong generalization claims

— viemccoy replied that the link to the paper's findings remained unclear.

Original post

OPJW#1460Jiaxin Wen|@JIAXINWEN22

@viemccoy i guess your read is because 1) it's learned early in pre-training 2) it transfers to instruct models my read is because models learn nothing deep early in pre-training, so this invalidates all "generalization" results

11:44 AM · May 22, 2026 View on X

REPLY𝚟⟢#1753𝚟𝚒𝚎 ⟢|@VIEMCCOY

@jiaxinwen22 I'm confused how this follows from the paper?

JWJiaxin Wen@jiaxinwen22

@viemccoy i guess your read is because 1) it's learned early in pre-training 2) it transfers to instruct models my read is because models learn nothing deep early in pre-training, so this invalidates all "generalization" results

6:44 PM · May 22, 2026 · 138 Views

7:10 PM · May 22, 2026 · 36 Views