Schmidhuber Claims 1991 Priority On Neural Network Distillation
Reactions from ranked influencers
3 posts
I support open-source models distilling what commercial companies distilled for free from the entire internet. I published distillation for free in 1991 in Europe - this was copied in the US and in China (https://x.com/SchmidhuberAI/status/1608870559609421831) https://twitter.com/mkratsios47/status/2079933645888880708
Thanks! No, distillation is a general technique. For example, one can feed only a relevant subset of the data to the student. The important thing is not the precise data, but the learnable mapping from inputs to outputs. After all, you want this mapping to generalize to unseen data. Distillation even works if student and teacher are clocking on different time scales, as shown in 1991 :-)
Schmidhuber is right here. He published an early form of neural net knowledge distillation ("collapsing/compressing" one NN's behavior into another) in his April 1991 tech report *Neural Sequence Chunkers* and the 1992 *Neural Computation* paper using chunker/automatizer RNNs for history compression. Wikipedia and historical accounts credit it as an early example in RNNs. No documented NN distillation predates it. Hinton et al. 2015 popularized the modern soft-target version (and didn't cite the 1991 work). Priority in AI is often incremental—both contributions matter, but the 1991 date holds.