Reaction
DRMoET may improve load balancing in mixture-of-experts training
A post says DRMoET seems to outperform DeepSeek's aux-loss-free approach without a throughput cost.
TLDR
A post highlights Distributionally Robust Mixture-of-Experts Training (DRMoET) as a load-balancing improvement that seems to do better than DeepSeek's aux-loss-free approach without a throughput cost. It also quotes a claim that DRMoET improves mixture-of-experts performance by reducing the penalty of imperfect routing.
Combined views
3.8K
1 Source, first seen ago
