Researchers Reformulate Layers as Bilevel Optimization Problems for Efficient Training

Our key idea is to reformulate the layer as a 𝐛𝐢𝐥𝐞𝐯𝐞𝐥 𝐨𝐩𝐭𝐢𝐦𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐩𝐫𝐨𝐛𝐥𝐞𝐦. We then construct an active-set Lagrangian “ghost” problem that preserves the local hypergradient, while reducing the backward computation to first-order operations. (3/n)