Modded-NanoGPT optimization result #29 (2026/05/11): @nilinabra has achieved a new step-count record of 2990 (40-step improvement) by halving the growth rate of the L2-norm of the hidden matrix parameters.
This result is better than the previous record with a p-value of 4e-5.
My thinking was to control weight norm without needing to tune weight decay. WD takes effect as the norms get near an equilibrium. Radial brake compresses the outward gradient component and takes effect immediately. It also affects the condition number differently than WD. https://nilin.github.io/radial-brake/
Modded-NanoGPT optimization result #29 (2026/05/11): @nilinabra has achieved a new step-count record of 2990 (40-step improvement) by halving the growth rate of the L2-norm of the hidden matrix parameters.
This result is better than the previous record with a p-value of 4e-5.
Modded-NanoGPT optimization result #29 (2026/05/11): @nilinabra has achieved a new step-count record of 2990 (40-step improvement) by halving the growth rate of the L2-norm of the hidden matrix parameters.
This result is better than the previous record with a p-value of 4e-5.