An LLM learning new skills by having weights removed sounds like a very confident use of the delete key. That’s the pitch from @Jacklu_me, who introduced Deletion Tuning in a November 10 post claiming that zeroing less than 0.1% of weights matches full fine-tuning’s accuracy “at scale.” Source
Full fine-tuning updates all of a model’s weights to adapt it to a task. The announcement’s text doesn’t name the models, tasks, benchmark scores or evaluation conditions behind its comparison. Source
@Jacklu_me also says the method is “thousands of times smaller” and “avoids forgetting,” calling it “a step toward continual learning.” But the post doesn’t define the size comparison or provide measurements of how well the model retains earlier capabilities. The size claim is not a license to shrink every number on the computing bill: it doesn’t establish a thousands-fold reduction in model size, training compute or inference costs. Source
On November 13, @xujustinj riffed on dropout, a technique used to train neural networks (NNs): “so instead of using dropout to train NNs, we can now use NNs to train dropout...” The quote-post called it an “insanely cool result” and congratulated Jack. Source
Comments