Reaction
Datology’s AI training recipe: “Merge, merge, merge”
Datology says it combined pre-extension models with post-extension checkpoints and used merging in reinforcement learning.
TLDR
During pretraining, Datology says it skipped learning-rate decay in favor of a running weight average. It then merged pre-context-extension models into post-extension checkpoints and also used merging in reinforcement learning. @willdepue endorsed the post: “based this looks right.”
Combined views
8.4K
2 Sources, first seen ago
Comments