• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
  • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    Muon's claimed distinctive fingerprint in model weights

    A user says Muon’s pattern differs from SGD and Adam, attributing the spectral signature to more aggressive updates in low-magnitude directions.

    SH
    1 Source, 17d ago, first seen 17d ago

    TLDR

    A user argues that inspecting the spectral distribution of a model’s hidden-layer weights can reveal whether Muon was used for training. The post describes a distinctive pattern in WᵀW—a matrix calculated from those weights—compared with SGD and Adam, and attributes it to Muon updating low-magnitude directions more aggressively.

    Combined views

    28.1K

    1 Source, first seen 17d ago

    Combined views

    28.1K

    1 Source, first seen 17d ago

    587 likes
    587 likes
    9 comments
    308 saves
    41 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    9 comments
    308 saves
    41 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    1 Source

    @hayou_soufianeIt’s easy to tell whether Muon was used to train a model just by looking at the empirical spectral distribution of its hidden layer weights. Compared with SGD/Adam, Muon leaves a very distinctive spectral signature in W^T W. This is because Muon updates low-magnitude directions more aggressively.

    1 Source

    @hayou_soufianeIt’s easy to tell whether Muon was used to train a model just by looking at the empirical spectral distribution of its hidden layer weights. Compared with SGD/Adam, Muon leaves a very distinctive spectral signature in W^T W. This is because Muon updates low-magnitude directions more aggressively.