Commentary on X
@MilesCranmer @kellerjordan0 Individual Muon orthogonalization per head is brilliant. Has that been done before?
@itchy_est I think they're cool as hell.
@Dorialexander Very interesting thanks for sharing
Commentary on X
@MilesCranmer @kellerjordan0 Individual Muon orthogonalization per head is brilliant. Has that been done before?
@itchy_est I think they're cool as hell.
@Dorialexander Very interesting thanks for sharing
View all
So K3, with one simple question: how do you even train a 3T parameters model? Even though the paper is not out, there is enough public information to get a relatively consistent picture. Just piling on experts won't make an economically viable model. What worked and made training and inference business profitable is highly sparse models with high-margin: initial investment in compute infrastructure is recouped through expert parallelism. Now as you get way beyond 1T parameters, you start running into physical limits of the available infrastructure. 896 experts won't fit on one node, even when only 16 are activated at generation time, you need to start splitting weights and latency starts to leak on all sides. First, K3 is a LatentMoE. The technique is not invented by Moonshot but by Nvidia (https://arxiv.org/pdf/2601.18089) and it's fundamentally a communication trick: instead of routing on the full hidden state, we down-projec, route, run the experts in that latent space, and up-project back. It's a fixed cost so that you can grow an indefinite amount of experts and still bound communication and bandwith. Now the critical part: Nvidia approach was focused on inference and Moonshot's approach is a *Stable* LatentMoE oriented toward training. Extreme sparsity means that each expert sees very few tokens per batch: this new stable implementation seems to fix the load-balancing estimator getting incredibly noisy as sparsity grows. Quantile balancing addresses a common MoE equilibrium issue that becomes explosive: some experts are too popular, too many token gets routed to them and then other experts just have to wait, idle and starving. K3's announcement explicitly said they anticipated expert paralllelism at inference time as a shaping training constraints. With current loading methods, we don't know how many tokens each expert got until they're routed, so buffers can't be sized ahead and you need a host round-trip to find out. Instead quantile loading maximizes total routing score subject to each expert taking exactly mk/n tokens. Roughly, each expert takes exactly its quota by constructio, then every buffer is known at compile time and the sync disappears. Attention Residuals seems to be fundamentally a depth trick already published by MoonShot but never validated at scale (the AttnRes paper only trains at 48B). Attention is pointed at layers instead of tokens. Normally every layer dumps its output onto a shared pile (the residual stream) and every later layer reads the whole pile back, everything added with equal weight. AttnRes lets each layer attend over the earlier layers and pull out what it actually wants. Same shape of deal as LatentMoE: a small fixed cost, under 5% overhead for about 25% of the gain. DeepSeek's mHC went in a similar direction by widening the residual stream until it is effectively doing attention across layers, except implicitly, through a chain of matrix products that can explode or collapse, which is why mHC needs sigmoids and doubly-stochastic normalization to hold it together. AttnRes doesn't fix that machinery, it avoids needing it: write the attention explicitly and softmax gives you the guarantees for free. We have much less information at this stage about the other new components (head muon, gated MLA, SiTU (though last one is likely needed for MXFP4?). Yet they all seem to proceed from the same philosophy shared by Su Jianlin in an April post (https://kexue.fm/archives/11729): "small-model-safe does not mean big-model-safe". Neither LatentMoE or Quantile Balancing made already much sense at smaller scale. They are becoming an integral part of the training and inference stack as we move up to an entirely different dimension of scaling and infrastructure deployment.