Good retweet for today given @tilderesearch's excellent post on https://blog.tilderesearch.com/blog/compositional-muon
rohan anil@_arohan_
Anisotropy is good for fwd and isotropy is good for bwd.
12:31 PM · Jun 5, 2026 · 7.2K Views
Tilde releases Compositional Muon, extending the Muon optimizer to composed transformer circuits via partner-whitened update rules
Sai Surya Duvvuri linked the optimizer's math to LoRA-RITE.
334465435486.3K
Sentiment
Users are excited about Tilde Research's Compositional Muon Optimizer for Transformers because it looks like a promising advancement and directly relevant to their need for a LoRA optimizer.
Pos
100.0%
Neg
0.0%
3 comments with sentiment.
Cluster Engagement
Posts from X
Most Activity
Most Activity
VIEWS17.5KBOOKMARKS59LIKES63RETWEETS7REPLIES3
Sai Surya Duvvuri@dvsaisurya
Just read this, nice research. We did something similar long back on LoRA factors.
https://arxiv.org/pdf/2410.20625
Tilde@tilderesearch
Introducing Compositional Muon, an optimizer that extends Muon from individual matrices to composed transformer circuits.
Modern optimizers usually draw trust regions around individual parameters. But in attention, the loss often sees compositions like QK^T and OV. Updating each factor independently can therefore control the wrong object. Compositional Muon closes this gap by deriving partner-whitened update rules. Each factor’s update is shaped by the spectral geometry of the matrix it is composed with, producing more stable composed updates and better effective learning-rate allocation across heads and layers.
For QK, this gives a head-local half-split rule. For OV, the circuit geometry selects a hybrid rule: (V) is optimized per-head, while (W_O) is optimized as the single matrix that aggregates all heads back into the residual stream.
CM improves over Muon at 340M and 1B scale, transfers to the modded-nanoGPT optimization benchmark, and can be approximated cheaply as partner-rescaled Muon via the isotropic rule.
The broader point is optimizer-architecture co-design: better optimizers should not only ask how to update a parameter, but what composed circuit that parameter participates in. CM is one step toward optimizers that respect the functional structure the loss actually sees.
14hViews 17.5KLikes 63Bookmarks 59
Tilde@tilderesearch
Read the full post here: https://blog.tilderesearch.com/blog/compositional-muon
1dViews 1.1KLikes 16Bookmarks 3
nor@norxornor
cool work! i was recently working on understanding the behavior of deep linear subnetwork reparameterization of linear layers - typical optimization with them (e.g. with weight decay) leads to something resembling a nuclear norm penalty on the weights (the visible spectral gap can be seen as an artifact of that) and also makes updates slower than they can be (bounds using single layer worst case twice are weaker than actual steepest descent bounds), and keeping in mind the factorized structure can help improve both, as in this approach
1dViews 578Likes 9
Andrew Rodriguez@andrewjrod
@tilderesearch Super cool!
1dViews 382Likes 3
ueaj@_ueaj
@dvsaisurya oh perfect we were looking for a lora optimizer actually cc @afrenkai
13hViews 79Likes 3
tom@thomasp_19
@tilderesearch tilde locked in
23hViews 221Likes 1
Oleg kAI@oleg_kai
@tilderesearch buried beat is composition-aware optimization is the missing layer between adam independence and full joint updates. current tensor cores resist it. software-hardware co-design becomes the training-cost bottleneck before architecture does.
21hViews 229
Guilherme O'Tina@guilhermeotina
the tension is real: adam updates Q and K independently but the loss only sees QK^T. gradient components that cancel in the product are invisible to the loss yet consume update budget. the practical question is whether jacobian recomposition per circuit block buys enough to justify the extra compute vs just tying lr ratios between paired matrices
19hViews 131
Xidulu@xidulu
@tilderesearch This reminds me of Quack from @laurence_ai and @benaibean : https://arxiv.org/abs/2511.21377
13hViews 16