Early submissions show a 260x speed gap between leaders.
In the same tradition of @srush_nlp if you can beat me, I'll tweet that you're strong kernel engineer https://www.gpumode.com/leaderboard/773?tab=rankings
Problem formulation and more details here https://www.gpumode.com/news/linear-algebra-kernels-age-of-research
Users are excited about the new kernel competition for batched QR decomposition on B200 because they find the classical problems fun, appreciate helpful framings like Flash Attention, and want to participate or bet on entries.
Is it done? 🚀
First two are a bit unbelievable.
Launching a new kernel competition: Linear Algebra Kernels For The Age Of Research.
First problem: batched QR decomposition on B200. Old math, modern hardware.
Prize: Rare swag and hangout in SF
This is what I mean by compute multipliers. People don't understand how powerful they are. Look at the multiple between the first one and the second one. 260 times. A single GPU getting shadow clone jutsud into 260 of them, and this is comparing it only to the runner up
Is it done? 🚀
First two are a bit unbelievable.
Launching a new kernel competition: Linear Algebra Kernels For The Age Of Research.
First problem: batched QR decomposition on B200. Old math, modern hardware.
Prize: Rare swag and hangout in SF
I have some mixed feelings about this result:
On the one hand, it's genuinely impressive. I didn't know that Shampoo could be configured to perform this well on the benchmark.
On the other hand, the way this performance boost was achieved seems difficult to call "Vanilla," for the following reason:
According to @_arohan_, the boost depends upon fixing a numerical linear algebra issue that he observed to occur in my initial standard DistributedShampoo run. He fixed the issue by enabling the flag rank_deficient_stability_config=PseudoInverseConfig().
Here's the problem: This is an undocumented flag. It is contained within the 12,000-line DistributedShampoo codebase, but it does not appear in any user-facing documentation.
As a result, if someone tries to train a model using DistributedShampoo without either (a) knowing about this special undocumented flag or (b) being prepared to detect and fix the numerical linear algebra issues that may occur without it, then they won't be able to achieve @_arohan_'s level of Shampoo performance. This level of effort would be considered atypical for mere hyperparameter tuning.
-- [Note on Muon baseline in plot below: Rohan's post compared Shampoo to a slightly undertuned Muon baseline from 2026/05/01, which reached the target loss in 3375 steps. This resulted in a 50-step gap between Shampoo and Muon. In the figure below I'm using the up-to-date 2026/05/03 baseline, which reaches the target in 3325 steps. This results in the step-counts exactly matching between Muon and the tuned/stabilized Shampoo variant.]
Just to be clear: submissions can be human + coding agents (.codex or claude) or harness over multiple models. You can access compute via modal.
If you are into linear algebra, this is a flex at designing hardware native algos (like Flash did Flash Attention)
If you are a kernels engineer, you can test if you can generalize to a different algorithm than MoE or Attention that demands different types of performance work.
Regardless, this is a very hard challenge.
LFG!
Nvidia GPUS must go faster for Linear Algebra Kernels. That shouldn’t be the bottleneck in the Age of Research.
Bring your best harness, human ingenuity, models, proto RSI to improve on top of CuSolver.
Problem formulation and more details here https://www.gpumode.com/news/linear-algebra-kernels-age-of-research
Imagine if you got this guy and spoonfeed him beta alanine and caffeine dry, with 6% salted milk to wash it down. And you had a hundred thousand clones of him. What would happen?
This is what I mean by compute multipliers. People don't understand how powerful they are. Look at the multiple between the first one and the second one. 260 times. A single GPU getting shadow clone jutsud into 260 of them, and this is comparing it only to the runner up
2128 to 1711 us today. 🤘
The Cuda Colonel still in top place but only by a small margin to nikhilbarhate99
Is it done? 🚀
First two are a bit unbelievable.
@wbrenton3 Free! https://github.com/gpu-mode/popcorn-cli/blob/main/docs/linalg-qr-b200.md
Imagine what happens when you start taping out your own chips, with hardware rasterizers, physics simulation circuits, hardware raytracers, and let these AIs learn in simulated environments at ludicrous speeds
@marksaroufim if you have a batched QR decomposition then the best optimization is to figure out how not to have that
@_arohan_ The strongest legit entry so far is by @blelbach
Is it done? 🚀
First two are a bit unbelievable.
@marksaroufim Omg shoutout to Trefethen and Bau! Math 571 at Michigan (https://www.coursicle.com/umich/courses/MATH/571/) killed me. Didn't realize it was relevant to optimizers...should have looked into this before getting too deep into GEMMs and Attention kernels :(
@marksaroufim Must have assumed that QR is not frontier level research 😃
lol Fable crushed my baseline
😱
i'm trying to spread a rumor that the prize is a wig made out of @MillionInt 's hair
i'd love your help
@theycallmeMohan These classical problems are significantly more fun. I hope you enjoy participating as much as we did designing
@_arohan_ time to finally get http://gpumode.com out of my bookmarks now that I get daily updates on my TL
I swear I read something about Jt’s hair strand on corp
i'm trying to spread a rumor that the prize is a wig made out of @MillionInt 's hair
i'd love your help
@marksaroufim Where does one get b200 capacity
@_arohan_ First 3 have caching issues, investigating (update coming for kguard for coverage) i have my current best #7 that is feasible
@marksaroufim @theycallmeMohan What if i told you I had 3x3 convolutional kernels I only spent a few hours on that are better than vanilla CUDNN on Blackwell, it's like they haven't prioritized much outside of basic transformer stuff.
