Moonshot AI's K3 Adds KDA, AttnRes, and LatentMoE for 2.5x Efficiency
Reactions from ranked influencers
2 posts
Still waiting on the tech report but a few innovations: 1/ KDA swaps Gated DeltaNet's single scalar decay for learned per-dimension forgetting. 2/ AttnRes retrieves selectively across depth. 3/ LatentMoE activates 16 of 896 experts, balanced from router score quantiles.
They target the spots where standard implementations use one-size-fits-all rules (uniform forgetting in linear attention, residuals weighting every layer equally, heuristic expert balancing).