Will DePue calls current era golden age for optimizers
Research engineer Will DePue described the present as a golden age for optimizers, with new variants appearing daily including SoapyShampooGluon and ongoing releases built on RMS matching. Research engineer Jerry Tworek predicted further expansion of progress ahead. Engineer Bilal compared the current pace of optimizer development to the environment faced by machine learning researchers in 2016.
@willdepue Golden age of golden ages ahead
golden age for optimizers right now. every day you see another SoapyShampooGluon ^-1/2 (RMSMatched) drops
golden age for optimizers right now. every day you see another SoapyShampooGluon ^-1/2 (RMSMatched) drops
@MillionInt one might call it the diamond age
@willdepue Golden age of golden ages ahead
Decoupled and Dualized with a side of fries
golden age for optimizers right now. every day you see another SoapyShampooGluon ^-1/2 (RMSMatched) drops
😭 btw one good thing of optimizer is that you don't need to worry about train-inference mismatch when changing it.
golden age for optimizers right now. every day you see another SoapyShampooGluon ^-1/2 (RMSMatched) drops
@willdepue so this is what it was like to be an ml researcher in 2016
golden age for optimizers right now. every day you see another SoapyShampooGluon ^-1/2 (RMSMatched) drops