Lucas Beyer noted a similar pattern with ResNets and ViTs.
I don’t know what the phenomena is called:
Sometimes the field mines improvements near a local neighborhood.
Like Adam -> (badam, dadam, madam), Shampoo -> Muon -> (Duon, Buon, Luon), last few made up instead of questioning whether the original formulation itself is the right question. You get so much math explaining these variants bordering slop. Same happened with Transformers too.
Mathematically sophisticated but solving the wrong problem.
Many users dismiss local variants of Adam and Shampoo as unoriginal spam and noise produced by groupthink and publish-or-perish incentives instead of addressing base assumptions.
Llms have made it easy to now spam methodological improvements that are largely bordering noise.
I don’t know what the phenomena is called:
Sometimes the field mines improvements near a local neighborhood.
Like Adam -> (badam, dadam, madam), Shampoo -> Muon -> (Duon, Buon, Luon), last few made up instead of questioning whether the original formulation itself is the right question. You get so much math explaining these variants bordering slop. Same happened with Transformers too.
Mathematically sophisticated but solving the wrong problem.
@_arohan_ I don't think it's new or llm related. We had the same with ResNets and later with ViTs, and those were before llms. It's just the easy research to do.
Llms have made it easy to now spam methodological improvements that are largely bordering noise.
@_arohan_ it’s difficult seeing [outside of] your box
I don’t know what the phenomena is called:
Sometimes the field mines improvements near a local neighborhood.
Like Adam -> (badam, dadam, madam), Shampoo -> Muon -> (Duon, Buon, Luon), last few made up instead of questioning whether the original formulation itself is the right question. You get so much math explaining these variants bordering slop. Same happened with Transformers too.
Mathematically sophisticated but solving the wrong problem.
@_arohan_ Thoughts on ademamix? I think it kinda missed the multiscale inductive bias but it was very close and very early
@_arohan_ GRPO variants from last year will say hi
@_arohan_ @nathancgy4 Oh man, I think I have a really good explanation for this, but it's a bit longer than a tweet. I might have to blog about this: I think it's because all components of neural network training have to work together, so it's hard to do non-local improvements.
@_arohan_ Wait, I think youre suggesting all those extensions are just post-hoc rationalizations pasted onto an original breakthrough?
@_arohan_ what an example!
@_arohan_ feels like naming satire just keeps becoming realer over time
not sure if we need new names or just to sit with the original ones longer
@_arohan_ "badam" (almond)
@_arohan_ real recognize real discovering meta-methods while ignoring base assumptions is how the loop keeps running
@_arohan_ groupthink + publish or perish
@_arohan_ yes so dissatisfying but apparently everything else is market irrelevant
@_arohan_ People show up for hacks, not first principles.
@_arohan_ greedy optimization is called local maxima. but honest question - is there a taxonomy for the "just keep changing letters" phase?
i sometimes wonder if we're just doing a local search over research programs. once the abstraction is fixed, you get high-sophistication perturbations: Adam→variants, Shampoo→variants, Transformer→variants. i wonder if it's even optimizing inside the right model class most of the time
@_arohan_ anyone who has followed AI research closely knows that AI slop has been around a lot longer than LLMs
@_arohan_ Some relevant advice from Hamming:
Lucas Beyer noted a similar pattern with ResNets and ViTs.
I don’t know what the phenomena is called:
Sometimes the field mines improvements near a local neighborhood.
Like Adam -> (badam, dadam, madam), Shampoo -> Muon -> (Duon, Buon, Luon), last few made up instead of questioning whether the original formulation itself is the right question. You get so much math explaining these variants bordering slop. Same happened with Transformers too.
Mathematically sophisticated but solving the wrong problem.