The poster calls the swing “mode-hopping” and says validation loss decreased smoothly throughout.
A post says OLMo3-32B’s arithmetic diagnostic accuracy reached 81% at 2.17 trillion pre-training tokens, fell to 0% at 2.19 trillion, then rose to 81.7% at 2.21 trillion. The poster calls it “mode-hopping” and says validation loss decreased smoothly throughout.
—
2 Sources, first seen ago
@che_shr_cat1/ During OLMo3-32B pre-training, accuracy on an arithmetic diagnostic hits 81% at 2.17T tokens, collapses to 0% at 2.19T, and jumps back to 81.7% at 2.21T. All while validation loss decreases smoothly. The phenomenon: mode-hopping. 🧵
@jiaxinwen22RT @che_shr_cat: 1/ During OLMo3-32B pre-training, accuracy on an arithmetic diagnostic hits 81% at 2.17T tokens, collapses to 0% at 2.19T,…@che_shr_cat1/ During OLMo3-32B pre-training, accuracy on an arithmetic diagnostic hits 81% at 2.17T tokens, collapses to 0% at 2.19T, and jumps back to 81.7% at 2.21T. All while validation loss decreases smoothly. The phenomenon: mode-hopping. 🧵@jiaxinwen22RT @che_shr_cat: 1/ During OLMo3-32B pre-training, accuracy on an arithmetic diagnostic hits 81% at 2.17T tokens, collapses to 0% at 2.19T,…@che_shr_cat1/ During OLMo3-32B pre-training, accuracy on an arithmetic diagnostic hits 81% at 2.17T tokens, collapses to 0% at 2.19T, and jumps back to 81.7% at 2.21T. All while validation loss decreases smoothly. The phenomenon: mode-hopping. 🧵@jiaxinwen22RT @che_shr_cat: 1/ During OLMo3-32B pre-training, accuracy on an arithmetic diagnostic hits 81% at 2.17T tokens, collapses to 0% at 2.19T,…@che_shr_cat1/ During OLMo3-32B pre-training, accuracy on an arithmetic diagnostic hits 81% at 2.17T tokens, collapses to 0% at 2.19T, and jumps back to 81.7% at 2.21T. All while validation loss decreases smoothly. The phenomenon: mode-hopping. 🧵@jiaxinwen22RT @che_shr_cat: 1/ During OLMo3-32B pre-training, accuracy on an arithmetic diagnostic hits 81% at 2.17T tokens, collapses to 0% at 2.19T,…—
Not ranked yet
—
Not ranked yet
