Alexander Doria prefers synthetic pretraining for controlled model behavior experiments.
What's the most valuable thing you can do with vintage LMs like Talkie? I think people are misled by Demis's pitch about rediscovering Relativity. Vintage LMs are just great baselines for LM science, letting you test many hypotheses about how pre-training and post-training interact.
Positive users like vintage language models for enabling controlled experiments on AI rediscovering science concepts, while negative users call some models too small and undertrained to be useful.
No Digg Deeper questions have been answered for this story yet.
@jiaxinwen22 I like vintage models a lot (likely trained the first one ever) but synthetic pretraining in general is a better frame for controlled experiments.
What's the most valuable thing you can do with vintage LMs like Talkie? I think people are misled by Demis's pitch about rediscovering Relativity. Vintage LMs are just great baselines for LM science, letting you test many hypotheses about how pre-training and post-training interact.
@jiaxinwen22 talkie is def too small and undertrained to do anything for meaningful related to ‘rediscovering science’
it doesn’t even recall/understand relativity
@aniketthh I'm not saying this because of today's performance of Talkie, but just in general vintage LMs -- even future really capable ones
@jiaxinwen22 what’s a hypothesis you’d like to test with one?
@jiaxinwen22 We did something similar here -
@jiaxinwen22 what do you mean by 'misled'? If we have the compute and all the data (not only text) until Einstein's time, rediscovering Relativity is a good way to test if our current training recipe can lead to real insightful AI.