Reactions from ranked influencers
2 posts
Turns out you can initialize an MLP with knowledge inside of it, no training required. Hazy research just showed that if you do that properly, a transformer can query that knowledge and use it! Continual learning???
MLPs store facts in language models. Can we write them into Transformers without training? New work w/ amazing team @garctrob @ronnygjunkins @EyubogluSabri, Atri Rudra & @HazyResearch gives a ✨closed-form✨ recipe for fact-storing, Transformer-ready MLPs. Accepted at COLM 2026!