Dan Fu, Together AI VP of Kernels, releases ParallelKernelBench to evaluate LLMs on writing multi-GPU kernels

Excited to release PKB Parallel Kernel Bench, led by Willy Chan and Nathan Paek @asplencmnt!!

A benchmark of mostly net-new multi-GPU kernel problems (solutions are independently useful for real-world workloads).

Fun fact - this is (my) first paper that I've posted on @askalphaxiv instead of arXiv. arXiv's moderation policies have become increasingly onerous (this one on hold for a month) - alphaXiv has a lot more features and is becoming my go-to!

Auto-blog https://www.alphaxiv.org/overview/2606.parallel-kernel-bench

Highlight - we built this benchmark to be largely net new multi-GPU problems. We provide the problem definition and reference PyTorch code, but no kernel solution to train against.

We were able to generate some new kernels that would have been entire papers during my PhD!

But why are multi-GPU kernels so different? Single-GPU kernels are bottlenecked by compute and memory bandwidth. Multi-GPU kernels by the interconnect.

Each PKB task hands the model a PyTorch + NCCL reference and asks it to communicate directly across GPUs via symmetric memory.

Frontier models struggle.

→ Best zero-shot: 28/87 correct, 22 beat the PyTorch + NCCL baseline → With 3 attempts: 36/87 correct, but fast1@3 tops out at 31%

Weak models fail to compile. Strong reasoners compile cleanly and return wrong answers.

Check out Nathan's @asplencmnt thread below or the Together for more details, including detailed error analysis of where models go wrong, and the relative frontier of where closed vs. open models are today.

@askalphaxiv PKB of course builds on great work from friends and collaborators - the original KernelBench from @simonguozirui and @anneouyang, as well as great work on the fundamentals like ParallelKittens by @stuart_sul.

Excited to push on this frontier!

okay a much more accurate picture

Check it out at https://github.com/togethercomputer/ParallelKernelBench

Check out the code for examples and details on running it: https://github.com/togethercomputer/ParallelKernelBench Blog: https://www.together.ai/blog/parallelkernelbench And paper: https://www.alphaxiv.org/abs/2606.parallel-kernel-bench

@simonguozirui @khshind thoughts on https://uccl-project.github.io/posts/commbench/ ?

LLMs write fast single-GPU kernels. Ask for a multi-GPU one and they fall apart.

ParallelKernelBench measures how they fail by benchmarking against 87 problems pulled from real codebases including Megatron-LM, DeepSpeed, DeepEP, TensorRT-LLM, NeMo-RL.

New research from Willy Chan @asplencmnt @simonguozirui @simran_s_arora and @realDanFu

That said, a few models did find solutions faster than the original repo code, producing net-new kernels! The example below speeds up a real vision workload.

Each problem starts from a standard PyTorch + NCCL implementation and a description of the hardware topology. We task LLMs with replacing that reference with CUDA that communicates directly across GPUs using symmetric memory.

We curated PKB’s 87 problems from open-source production repos such as Megatron-LM, DeepEP, TensorRT-LLM, NeMo-RL, with wide coverage over parallelism types — TP, DP, CP, EP, FSDP/ZeRO, etc, and combinations of the above.

@realDanFu @askalphaxiv cc @tdietterich

We evaluated frontier LLMs and found they struggle: single-shot correctness plateaus at 32%, and only 25% of cases beat an unoverlapped PyTorch+NCCL baseline.

@simonguozirui @realDanFu @khshind You don’t need to write CUDA to write Ring attention though

Struggling to write Ring Attention on TPUs/GPUs with @khshind was one of the original motivations for KernelBench 😅

It feels full circle with ParallelKernelBench — a dedicated eval to see whether LLMs can write fast multi-GPU kernels 📡

Introducing the latest KernelBench family member: PKB, led by awesome undergrad researchers @opengroundsFX & @NathanPaek9368! (+ the always amazing @simran_s_arora @realDanFu for their guidance 🙏)

An agentic loop (compile, test, profile, revise) helps. Gemini 3 Pro went from 24 to 35/87 correct, then plateaued after ~20 steps.

Feedback fixes syntax, not rank coordination, collective ordering, or transfer-mechanism choice. TMA and NVLS stay almost unused.

Super grateful to my coauthor Willy Chan, collaborator @simonguozirui, and awesome mentors @simran_s_arora and @realDanFu! And huge thanks to @togethercompute for making the project happen! Check out PKB here:

Blog 🌐: https://www.together.ai/blog/parallelkernelbench Paper 📜: https://www.alphaxiv.org/abs/2606.parallel-kernel-bench GitHub 💻: https://github.com/togethercomputer/ParallelKernelBench HuggingFace 🤗: https://huggingface.co/datasets/togethercomputer/ParallelKernelBench_Problems