CoreAutoAI co-founder Rohan Anil resurfaces a 1971 regularization method that stabilizes the Shampoo optimizer against NanoGPT training collapse

@bozavlado @_arohan_ FYI a useful conversion formula in the benchmark is 0.0045 loss ≈ 100 steps

Funny thing is, after you fix the pseudoinverse in Shampoo (thx to @_arohan_ ), then the actual exponent in the (double-sided) Shampoo does not matter much in the @kellerjordan0 NanoGPT benchmark. You can do -1/4, -1/2, even -1/5 and get apx. same results.

(And without that pseudoinverse fix everything is just trash).

To be clear, I used momenta 0.95/0.99 and forced update of each matrix to have the same Frobenius norm.

@_arohan_ Uphill steps and geodesic acceleration

@_arohan_ thinking about the fletcher / marquardt stuff between rides is such a weird flex

really is the best way to get ideas though

@bozavlado @_arohan_ @kellerjordan0 adam's epsilon gets similarly boring after you decouple weight decay, just a numerical bumper at that point