Positive users appreciate Horace He's clarification on RMSNorm and LayerNorm performance nuances for highlighting fusion, while negative users feel it glosses over memory-bound bottlenecks in deep-dives.
Based on 3 visible X reactions from 9 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@cHHillee had you dropped another banger for me to read i would not have made this mistake; ) was only thinking about it from the perspective of each standalone kernel... but makes sense given only so many registers/smem a kernel can have https://x.com/waterloo_intern/status/2076401888678043814/photo/1
@cHHillee Interesting to see how memory-bound bottlenecks keep getting glossed over in these deep-dives—practical perf optimization is where the real conversation should be.
@cHHillee Yup thanks for mentioning fusion, usually ppl ignore that perspective
I agree with the quoted poster, but I do think the situation is a bit of the bell curve/midwit tweet. The original paper didn't really understand memory-bound kernels, and had some very incorrect assertions on performance. On the other hand, once you start really pushing performance (like matmul epilogues), the additional state tracking needed for the mean shift makes it somewhat more painful to fuse. So yes, layernorm is not 2x more expensive than rmsnorm. But it's not "free" either.
LayerNorm's mean calculation makes fusion into matmul epilogues more difficult
@cHHillee had you dropped another banger for me to read i would not have made this mistake; ) was only thinking about it from the perspective of each standalone kernel... but makes sense given only so many registers/smem a kernel can have https://x.com/waterloo_intern/status/2076401888678043814/photo/1
@cHHillee Interesting to see how memory-bound bottlenecks keep getting glossed over in these deep-dives—practical perf optimization is where the real conversation should be.
I agree with the quoted poster, but I do think the situation is a bit of the bell curve/midwit tweet. The original paper didn't really understand memory-bound kernels, and had some very incorrect assertions on performance. On the other hand, once you start really pushing performance (like matmul epilogues), the additional state tracking needed for the mean shift makes it somewhat more painful to fuse. So yes, layernorm is not 2x more expensive than rmsnorm. But it's not "free" either.
Positive users appreciate Horace He's clarification on RMSNorm and LayerNorm performance nuances for highlighting fusion, while negative users feel it glosses over memory-bound bottlenecks in deep-dives.
Based on 3 visible X reactions from 9 accounts; directional sample.
Ask a question below.
Published answers will appear here.