Transformer pruning could adapt one model to different compute budgets without retraining

A collaborator says the method learns one importance ranking of MLP neurons and attention heads with the model’s weights frozen, then drops the least important units at inference to meet a compute budget without retraining. Across 27 vision and text encoders spanning 22M to 8B parameters, they report that performance degrades smoothly up to 60% sparsity. In one example, they say pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters with a 0.8% drop in ImageNet-1k linear probing.