EPFL MLO Lab introduces Magnitude-Direction Decoupling to improve neural network training efficiency at scale

@giffmana Yeah, their Rotational Equilibrium work was also fantastic - one of the few papers which feels optimized for being honest and actually helping the reader understand the work.

@giffmana Actually, this idea is well supported by the weightwatcher theory. It is saying that is the shape, not the scale, of W that really modulates the generalization accuracy.

This is the natural continuation of this group's previous several works on optimization, and I really like their style.

A lot of experiments to look at all possible details. As opposed to a wall of theory and then one single experiment with untuned baselines.

@giffmana The world if every ML paper were like this

@filipviz Yep fully agree

@giffmana Refreshing to see actual data instead of just more whitepaper fluff.

@giffmana The focus on exhaustive ablation over theoretical posturing is exactly what moves the field forward. Reproducibility starts with showing your work at every marginal gain.