A 1991 paper's claimed role in three techniques used by modern language models

Jürgen Schmidhuber says his 1991 paper, “Neural sequence chunkers,” introduced three techniques he considers essential to modern large language models: pre-training for deep neural networks, positional encoding and distilling knowledge from one neural network to another. His overview describes networks trained to predict their next input, passing unexpected inputs to higher levels.