Strata@ChainZenitTECH
@SchmidhuberAI @CadeMetz that is frustrating, have you reached out to them?
@SchmidhuberAI @CadeMetz @giffmana 👀👀👀 read and learn!!
A recent NYT article by @CadeMetz [1] claims that neural network distillation was first developed in 2015 by a team of Google researchers including G. Hinton [2]. Not true! I published the technique in 1991 [3]. [2] failed to cite [3]. See the overviews [4][6]. The NYT article [1] states: "... distillation is an effort to copy the behaviour of the system, as opposed to copying the text verbatim." That's right! Distillation enables non-trivial forms of plagiarism [5]. Annotated References (easy to find on the web): [1] Cade Metz, New York Times (6 July 2026). "American AI companies Say Chinese Copycats are Quickly Catching Up" [2] O. Vinyals, J. A. Dean, G. E. Hinton (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531. [2] did not cite the 1991 neural network distillation procedure [3]. [3] J. Schmidhuber (JS). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992. Based on TR FKI-148-91, TUM, 1991. See Section 4 on the "conscious" chunker and a "subconscious" automatiser which introduced a general principle for transferring the knowledge from one neural net (NN) to another. Suppose a teacher NN has learned to predict (conditional expectations of) data, given other data. Its knowledge can be compressed into a student NN, by training the student NN to imitate the behavior of the teacher NN (while also re-training the student NN on previously learned skills such that it does not forget them). In 1991, I called this "collapsing" or "compressing" one NN into another. Today, this is widely used, and also referred to as "distilling" or "cloning" the behavior of a teacher NN into that of a student NN. It even works when the NNs are recurrent and operate on different time scales. [4] JS. Who invented knowledge distillation with artificial neural networks? Technical Note IDSIA-12-25, IDSIA, Nov 2025. https://people.idsia.ch/~juergen/who-invented-knowledge-distillation-with-neural-networks.html [5] JS. How 3 Turing awardees republished key methods and ideas whose creators they failed to credit. Technical Report IDSIA-23-23, Swiss AI Lab IDSIA, 2023 (updated 2025). [6] @hardmaru & JS (2026). Munich 1991: the Roots of the Current AI Boom. With a preface by David Ha.
@CadeMetz Related: https://x.com/SchmidhuberAI/status/1844022724328394780
Ask a question below.
Published answers will appear here.
