4h ago

Tilde introduces Aurora optimizer to correct neuron death in Muon updates

155032.9K

——0——

Tilde published a technical post introducing Aurora, a leverage-aware optimizer for rectangular matrices. Aurora fixes persistent small updates to some MLP neurons in Muon and NorMuon by redistributing mass equally across rows of up- and gate-projection updates while keeping exact orthogonalization. At 1B scale Aurora delivered clear gains over both prior methods and set a new state-of-the-art on the modded-nanoGPT speedrun. Research engineers shared the post and updates on social media.

Original post