He cited Adam derivatives and Muon as key examples.
I don’t know what the phenomena is called:
Sometimes the field mines improvements near a local neighborhood.
Like Adam -> (badam, dadam, madam), Shampoo -> Muon -> (Duon, Buon, Luon), last few made up instead of questioning whether the original formulation itself is the right question. You get so much math explaining these variants bordering slop. Same happened with Transformers too.
Mathematically sophisticated but solving the wrong problem.
Users criticized local variants of Adam and Shampoo as methodological spam and noise produced by groupthink and publish-or-perish pressures instead of addressing core problems.
Llms have made it easy to now spam methodological improvements that are largely bordering noise.
@_arohan_ I don't think it's new or llm related. We had the same with ResNets and later with ViTs, and those were before llms. It's just the easy research to do.
@_arohan_ Some relevant advice from Hamming:
@_arohan_ Thoughts on ademamix? I think it kinda missed the multiscale inductive bias but it was very close and very early
i sometimes wonder if we're just doing a local search over research programs. once the abstraction is fixed, you get high-sophistication perturbations: Adam→variants, Shampoo→variants, Transformer→variants. i wonder if it's even optimizing inside the right model class most of the time
@_arohan_ it’s difficult seeing [outside of] your box
I don’t know what the phenomena is called:
Sometimes the field mines improvements near a local neighborhood.
Like Adam -> (badam, dadam, madam), Shampoo -> Muon -> (Duon, Buon, Luon), last few made up instead of questioning whether the original formulation itself is the right question. You get so much math explaining these variants bordering slop. Same happened with Transformers too.
Mathematically sophisticated but solving the wrong problem.
@_arohan_ @nathancgy4 Oh man, I think I have a really good explanation for this, but it's a bit longer than a tweet. I might have to blog about this: I think it's because all components of neural network training have to work together, so it's hard to do non-local improvements.
@_arohan_ There's a name for it: papers are optimized by gradient descent too — reviewers can only score "differential from X", never "is X the right question." So Adam→madam variants are literally the equilibrium of that loss surface, not a failure of taste.
@_arohan_ GRPO variants from last year will say hi
@_arohan_ anyone who has followed AI research closely knows that AI slop has been around a lot longer than LLMs
@_arohan_ feels like naming satire just keeps becoming realer over time
not sure if we need new names or just to sit with the original ones longer
@_arohan_ groupthink + publish or perish
@_arohan_ Wait, I think youre suggesting all those extensions are just post-hoc rationalizations pasted onto an original breakthrough?
@_arohan_ greedy optimization is called local maxima. but honest question - is there a taxonomy for the "just keep changing letters" phase?
@_arohan_ real recognize real discovering meta-methods while ignoring base assumptions is how the loop keeps running
@_arohan_ what an example!
@_arohan_ "badam" (almond)
@_arohan_ People show up for hacks, not first principles.
@_arohan_ yes so dissatisfying but apparently everything else is market irrelevant
@giffmana @_arohan_ Where do you think LLMs got it from
He cited Adam derivatives and Muon as key examples.
I don’t know what the phenomena is called:
Sometimes the field mines improvements near a local neighborhood.
Like Adam -> (badam, dadam, madam), Shampoo -> Muon -> (Duon, Buon, Luon), last few made up instead of questioning whether the original formulation itself is the right question. You get so much math explaining these variants bordering slop. Same happened with Transformers too.
Mathematically sophisticated but solving the wrong problem.