8/ Finally, a complementary direction: we tested compatibility with modularity-aware pretraining (EMO, https://arxiv.org/abs/2605.06663).

Modularity-aware pretraining gives a +3.6pp lift and ~87× lower pre-distillation PPL on the dense student model vs. a regular-MoE teacher.

We see such compression-aware pretraining as a co-design direction worth exploring.

Thanks to @sewon__min for introducing the work at ICLR, and to the authors @RyanYixiang @AkshitaB93 for the great work!

2/ The structure of MoE makes this natural.

Per-expert computations are independent until the weighted sum. Concatenating their weights into a dense FFN preserves intermediate activations.

The problem comes down to which experts give the best dense FFN init for distillation.

5/ Across 350 configurations on Qwen3-30B-A3B, a clear pattern emerged:

diversity-aware selection (DO-ACP) with no merging (pure pruning) consistently wins after distillation.

The pattern holds on DeepSeek and GPT-OSS MoE models too.

4/ Our intuition is that output diversity should matter for selection.

Drawing inspiration from the D-Optimal criterion in experimental design, we introduce a diversity-aware scoring metric (DO-ACP) and compare with other expert scoring metrics.

We set up a pipeline that decouples these choices [number of experts / scoring / grouping / magnitude scaling] for systematic investigation.

7/ Benchmark numbers alone don’t tell the whole story, so we also conducted qualitative analysis.

MoE→dense (DO-ACP) wins over dense→dense (D2D) on two fronts: it is more often fluent and gets more facts right.

More details and examples in the paper!

3/ The design space is wider than it looks.

E.g., from 128 experts, we can pick 8 and concatenate, or pick 32 and merge them into 8 groups of 4, etc. Both scoring and grouping metrics have multiple candidates from the expert-pruning/merging literature.

6/ How does our best MoE→dense recipe compare to just pruning a dense model directly?

Surprisingly, at matched total params for teacher and student, our MoE→dense (DO-ACP) outperforms dense→dense (D2D) pruning by +6.3pp avg accuracy at ~1.6× faster training wall-clock.

Please check out the paper for more details 🙂

Code: http://github.com/krafton-ai/moe-to-dense Paper: http://arxiv.org/abs/2605.28207

This is joint work with amazing collaborators: @JihunYun_ai, Haechan, @gmkim_ai, Joonghyun, @jaewoong_cho from @Krafton_AI

@jhyuckkim does the 87x pre-distillation PPL gap actually survive fine-tuning? i've had big ppl drops vanish after one epoch