Report
A transformer reportedly keeps learning beyond a predicted stability threshold
A 35-million-parameter transformer reportedly kept lowering its loss after sharpness crossed a predicted stability threshold in a CS 312 experiment.
TLDR
A CS 312 writer reports that a 35-million-parameter transformer crossed a predicted sharpness threshold of 200 at update 375. Its median sharpness then reached 260 while its loss decreased overall. Sharpness measures the steepest curvature of the loss function. The writer says simplified models predict instability beyond the threshold, but why this run stabilized so far above it remains unclear.
Combined views
13.9K
3 Sources, first seen ago