Researcher Andreas Kirsch says upcoming Mythos and Fable evaluations will test the theory
@Jsevillamol Seeing the evals for Mythos/Fable will be interesting
My pet theory is that the driving force of improvements for out-of-distribution tasks is mostly pure pretraining scaling, which if you squint this is broadly consistent with.
No Digg Deeper questions have been answered for this story yet.
@Jsevillamol I have two ideas: - larger/deeper models have better in-context-learning - simply more, and more diverse training data
My pet theory is that the driving force of improvements for out-of-distribution tasks is mostly pure pretraining scaling, which if you squint this is broadly consistent with.
@Jsevillamol You will test Mythos ?
@Jsevillamol Say more?
@Jsevillamol What effect do you think this belief has on your timelines (in a sense that "timelines" is meaningful)? If you thought it came from post-training, would this change your timelines/predictions
@Jsevillamol It will be a good hint for your take.