A new preprint by Nicolas Dufour, Elouan Gardès, Amil Dravid, Patrick Pérez and Loic Landrieu challenges the proposed explanation for Neural Thickets' reported gains. The authors say adding random biases to a model's output logits matched or exceeded perturbations to all its weights on most reasoning tasks they tested, even though the model's weights stayed frozen.
The Neural Thickets hypothesis, as the preprint describes it, proposes that pretrained language models have nearby task-specific experts that random weight changes can uncover. Kyutai describes the procedure as perturbing every weight, selecting the copies that score best on a small labeled set and having them vote. The preprint's authors say they reproduced the reported gains, but argue that the gains do not require the proposed weight-space explanation.
Their alternative, called Perturb-Select-Vote, generates varied predictors, chooses high-performing ones using a small validation set and combines their predictions. The version at the center of the comparison adds a random bias to output logits without changing its weights. The abstract suggests that output diversity, selection and voting explain the gains. It also reports improved performance with the logit-based approach on vision tasks under domain shift.
Dufour called Neural Thickets "sampling in disguise" in an Oct. 30 post. His shorthand, "same gains," is broader than the preprint's comparison, which specifies most reasoning tasks. The abstract does not give a task-by-task breakdown of that comparison.
Comments