Model run reportedly had no spikes amid four changes
A reply says attention-gate weight decay was added over concerns the model wasn't using early layers enough, while PDL was disabled in QuACK grouped GEMMs because of hangs.
TLDR
Asked whether different colors indicated that the run had spiked and resumed, a participant replied: “Nope, no spikes!” They listed four changes: at about 58k, adding attention-gate weight decay because of concerns the model wasn't using early layers enough; at about 82k, adding a faster EP kernel; at about 108k, changing the data mix a bit; and at about 109k, disabling PDL in QuACK grouped GEMMs because of hangs.
Combined views
971
4 Sources, first seen ago