So, we spent some time banging on the details for sparse MoEs, and here is my current understanding. Putting it out here if anyone is interested, and to see if this seems right.
I am actually kind of super CODA pilled right now https://arxiv.org/abs/2605.19269 ... need to work out the details for sparse MoEs though
Users like the CODA architecture's sparse MoE integration because its maximalism simplifies optimization problems and lets metrics be computed for free in epilogues, calling the approach promising.
No Digg Deeper questions have been answered for this story yet.
When we change the dense transformer into an MoE in the paper, the affected parts of the model we have circled in red. Additionally, we have to also add the router, dispatch and combine in most MoE training setups.
So, we spent some time banging on the details for sparse MoEs, and here is my current understanding. Putting it out here if anyone is interested, and to see if this seems right.
The most immediate consequence is that the RMSNorm needs to now be fused into a non-expert matmul and an expert matmul! As a reminder, the CODA RMSNorm trick *commutes* the scaling by statistic across the second GEMM so it can be put in an epilogue (not pictured below).
When we change the dense transformer into an MoE in the paper, the affected parts of the model we have circled in red. Additionally, we have to also add the router, dispatch and combine in most MoE training setups.
I like CODA maximalism for several reasons. (1) It simplifies the optimization problem. What's the best fusions to pick? Doesn't matter: everything goes in an epilogue. (2) Doing low precision/quantization? You may need some delicate conversions--CODA makes them free.
Similar analysis can be applied to the other epilogues straddling the dispatch-combine boundary!
(3) Computing metrics? Shove them in epilogues where you have spare compute, get them for free.
I like CODA maximalism for several reasons. (1) It simplifies the optimization problem. What's the best fusions to pick? Doesn't matter: everything goes in an epilogue. (2) Doing low precision/quantization? You may need some delicate conversions--CODA makes them free.
When we look at a standard MoE, we can see the result of RMSNorm 2 (yellow) is fed into two matmuls: the router (non-expert) and the first feed forward linear (expert). We treat the r-scale epilogue as "free" and so we don't mind duplicating it.
The most immediate consequence is that the RMSNorm needs to now be fused into a non-expert matmul and an expert matmul! As a reminder, the CODA RMSNorm trick *commutes* the scaling by statistic across the second GEMM so it can be put in an epilogue (not pictured below).
The easy fusion is the router: it's just a normal GEMM right after the RMSNorm, and we can fuse the r-scale right before the top-k. But the linear is tricky: to fuse the r-scale there, we must *communicate* the r-scale to the experts alongside the token dispatch.
When we look at a standard MoE, we can see the result of RMSNorm 2 (yellow) is fed into two matmuls: the router (non-expert) and the first feed forward linear (expert). We treat the r-scale epilogue as "free" and so we don't mind duplicating it.
This probably is the right thing to do in some situations? (1) In a Blackwell NVL72 regime, NVLink is pretty fast and you can easily fuse the extra comms into your symmetric memory kernel for dispatch. If you're using a NCCL all_to_all this is significantly more cursed.
The easy fusion is the router: it's just a normal GEMM right after the RMSNorm, and we can fuse the r-scale right before the top-k. But the linear is tricky: to fuse the r-scale there, we must *communicate* the r-scale to the experts alongside the token dispatch.
(2) While you /could/ instead fuse the r-scale into the communication kernel instead (where it can also be done for free), there is a problem: the backwards RMSNorm also needs to be fused into the backwards Linear... and you need the r-scale for this!
This probably is the right thing to do in some situations? (1) In a Blackwell NVL72 regime, NVLink is pretty fast and you can easily fuse the extra comms into your symmetric memory kernel for dispatch. If you're using a NCCL all_to_all this is significantly more cursed.
Similar analysis can be applied to the other epilogues straddling the dispatch-combine boundary!
(2) While you /could/ instead fuse the r-scale into the communication kernel instead (where it can also be done for free), there is a problem: the backwards RMSNorm also needs to be fused into the backwards Linear... and you need the r-scale for this!
Now, disclaimer, I haven't implemented any of this :) So this could all not actually work in practice. But it definitely seems like a promising direction!