Elliot Arledge's new KernelBench benchmarks evaluate CUDA megakernel generation, with Claude Opus 4.8 leading performance on NVIDIA B200 GPUs

by the way, "DNF" is not "it categorically cannot write the kernel", it's "Elliot got rate limited". GLM 5.2 is at the frontier in kernel engineering, simple as.

All 172 agent traces are public and render in HuggingFace's agent trace viewer: every message, tool call, and reasoning for each model writing a real kernel.

Mega: http://hf.co/datasets/Infatoshi/kernelbench-mega-traces Hard: http://hf.co/datasets/Infatoshi/kernelbench-hard-traces Official website: http://kernelbench.com

KimiDeltaAttention: torch.compile can't fuse the chunk-parallel delta-rule recurrence (yielding 0.07% of roofline perf), so a hand-written CUTLASS kernel dominates. Opus 4.8 reaches 80x over compile on the RTX PRO 6000.

Beyond the megakernel, a 6-problem hard CUDA/Triton deck. Speedup is over torch.compile (a strong baseline, not naive PyTorch). Paged attention is where compile falls down and a real kernel runs away with it: Opus 4.8 hits 56.8x on B200.

@LLMJunky i always recommend my best two courses as they are designed for complete beginners: https://www.youtube.com/watch?v=UU1WVnMk4E8&t=4913s https://www.youtube.com/watch?v=86FAWCzIe_4&t=25078s

i do recommend the first before the second as the second gets into kernels (the very point of this benchmark)

Top-K (bitonic) is the honesty check: these kernels are so small they're dominated by launch overhead, so the ceiling is low and speedups stay modest. Grading against the hardware roofline makes that obvious instead of hiding it.

@teortaxesTex Ugh why do we report infra issues as scores

Sonic MoE SwiGLU: grouped GEMM with a fused SwiGLU up-projection. GPT-5.5 and Opus 4.8 lead at ~6x over torch.compile.

FP8 GEMM is the opposite story: torch.compile already emits fp8 tensor-core GEMMs, so headroom is small. Everyone clusters around 2x, GPT-5.5 tops it at 2.5x. A benchmark where the baseline is genuinely hard to beat.

@teortaxesTex got rate limited. i figured id put these out now so people know about it rather than wait

W4A16 GEMM: int4 weight-only decode, the bandwidth-bound path that powers quantized inference. Solid speedups across the board.

@elliotarledge Why "DNF"? It strictly can't?

@xeophon didn't you say Fable should get 0 for fallbacks? or what that xeophon

@elliotarledge This is amazing work man, thanks so much!

Where can I learn more?

@elliotarledge @grok please give me 80/20 of what I can learn from this and take away to improve my workflow

@elliotarledge Cool

@elliotarledge Opus 4.8 getting slower on B200 is painful but makes sense. Most models write CUDA that ignores register pressure and SM layout on Blackwell. Composer 2.5 and DeepSeek actually scaling there is wild.

80/20 from KernelBench-Mega:

Claude Opus 4.8 autonomously writes megakernels hitting up to 19.4x speedup on B200 (strong on H100/RTX too). GLM-5.2 leads open models. 172 traces open-sourced.

Workflow upgrades: - Use frontier models in agent loops to generate fused kernels first, not last. - Study the traces to copy winning iteration patterns and prompting. - Benchmark on your target GPUs—gains are hardware-specific.

Prototype more ideas, faster.

@teortaxesTex will rerun overnight when rate limits are cleared. i abused the fuck out of glm 5.2 today

@elliotarledge you the real mvp