Together Compiler’s builders shared lessons from their optimization work on November 16, arguing that packing an entire model into one GPU kernel isn’t always the best target. Their alternative: choose which operations belong together, tune those groups, and use agents to explore the possibilities. Source Source Source
@dylan__lim wrote, “Last Year We Built Megakernels. This Year We Deleted Them.” The same post described whole-model megakernels as a “helpful abstraction” that can overcomplicate optimization. Source
Torch Compile gets surprisingly close
According to @dylan__lim, a tuned Torch Compile setup reached 98% of the team’s previous Llama 1B megakernel’s performance, using default Inductor GPU kernels alongside other optimizations. Source
@dylan__lim’s post doesn’t specify the hardware, batch size, performance metric or measurement protocol needed to independently assess that figure. Source
Torch Compile doesn’t receive a free pass, either. The same thread argues that its heuristics cannot reliably select the best implementation across workloads, and that measuring a kernel in isolation doesn’t reflect the speed of the complete workload. Source
Fusion needs to earn its keep
A whole-model megakernel combines the model’s operations into one GPU kernel. The catch, according to the builders, is that those operations must share the same resources. That can work well in some circumstances while limiting performance in others. One giant kernel also means one shared set of compromises. Source
“Fusion can be harmful,” @simran_s_arora wrote, arguing against blindly pursuing full-model megakernels. Source
The team instead favors identifying the right groups of operations to couple and tune. That creates more combinations to investigate. The builders believe agents are increasingly suited to exploring that larger search space and describe Together Compiler as an effort to demonstrate the idea. Source
Comments