Study finds pruning larger LLMs outperforms training smaller models from scratch, even with extra training tokens

Years ago, in 2018, we pushed back on the hype around network pruning in the ConvNet era.

The question is only bigger now that everyone wants small LLMs.

You want a strong small LLM. Would you start small — or inherit from something bigger?

📄 New paper: Small LLMs: Pruning vs. Training from Scratch

We find that pruning is more than a better initialization: simply giving randomly initialized LLMs more training tokens is often not enough to catch up.

🔍 Pruning outperforms training from scratch when training token budget is equal.

We pretrain a Llama3-8B model on 200B tokens, prune it, and train the small LLM on 50B tokens.

Comparing with a small LLM with same architecture and 50B training token, pruning initialized models always perform better.

We open-source our collection of all pruning methods implementation, GPU & TPU training and evaluation code.

Thanks @TaimingLu, Kunjun Li, @JiachenAI, @_mingjiesun, and @liuzhuang1234 for the support!

arXiv: http://arxiv.org/abs/2606.14150 code: http://github.com/zlab-princeton/pruning-vs-scratch

🔬 We apply 6 pruning methods across 3 different granularities: depth (Minitron-depth), width (Minitron-width, FLAP, Sheared Llama), and sparse pruning (SparseGPT, Wanda).

We compare the performance of initializing by pruning vs training from scratch with equal training tokens.

The paper from back then: "Rethinking the Value of Network Pruning" (ICLR 2019)

http://arxiv.org/abs/1810.05270

Same question, LLM now.

Recommendations:

💡 With a large pretrained model in hand and a limited training token budget, pruning is better than training from scratch. 💡 When the training budget is not limited, training from scratch can be competitive for coarser pruning, so a large pretrained parent is not always necessary

🔍 As pruning ratio increases, the advantage of depth pruning diminishes and eventually vanishes, while the advantage of width pruning is stable.

Depth pruning is sensitive to pruning ratio, while width pruning is robust.

🔍 Sometimes the advantage of pruning cannot be bridged with more training tokens.

When models are the trained from scratch with the whole token budget used in pretraining-pruning-retraining pipeline, it still underperforms methods with finer granularities.

This suggests that pruning is more than data transfer; it transfers the advantages of learning at a larger scale.

🔍 Across the structured pruning methods we examined, the advantage of pruning over training from scratch diminishes as training token budget increases, but the gap does not close.

Thanks for sharing this paper - very useful for the community! Curious if you’ve tried pruning during training of the larger model. I ran some experiments asking whether this can beat training the small model from scratch under the same end-to-end compute/token budget. I couldn’t get it to beat the naive scratch baseline, so I’d be curious to hear your thoughts.

@Zephyr271828 How does it compare to distilling?

@Zephyr271828 @0xSero

@Zephyr271828 Nice to see such a great paper!!! The results are as intuition and empirical results point to. Makes me remember of the ModernBERT recipe if I'm not mistaken

@Zephyr271828 Good framing for LLM research work. The practical part is not just trying a stronger model, but logging baselines, data splits, task-specific metrics, and failure cases so the result is reproducible.

@Zephyr271828 @TaimingLu @JiachenAI @_mingjiesun @liuzhuang1234 Really interesting work, though your repo link 404s currently

@Zephyr271828 @TaimingLu @JiachenAI @_mingjiesun @liuzhuang1234 Someone really needs to prune the GLM 5.2 so that it only focuses on programming related tasks and gets rid of everything else. I’m really growing tired of LLMs pretending to know everything models.

@Zephyr271828 @0xSero

@Zephyr271828 Try this. Check my page , I shared a great paper I think will help.

@liuzhuang1234 2018 was ConvNet compression. LLM pruning inherits structure that scratch has to rediscover. equal-cost gap confirms it.