Jürgen Schmidhuber argues his Munich team's 1991 papers introduced the foundational concepts for modern transformers, network distillation, and pre-training

VIEWS8.2KBOOKMARKS11LIKES50RETWEETS10REPLIES5

In 1991, the foundations for Transformers, Pre-training, Distillation, and World Models were already being built. These helped shape my own thinking, from my time at Google Brain to our Recursive Self-Improvement (RSI) work at @SakanaAILabs today. 🧠🗼

https://people.idsia.ch/~juergen/ai-boom-roots-munich-1991.html 👇

Jürgen Schmidhuber@SchmidhuberAI

I am proud of the work my team did in Munich in 1991, when compute was millions of times more expensive. We published the roots of today's trillion-dollar AI boom:

★ 3/1991: the first kind of Transformer (see the T in ChatGPT) - now called the unnormalized linear Transformer: the predecessor of the normalized quadratic Transformer

★ 4/1991: Pre-Training (the P in ChatGPT) & Neural Net Distillation (see DeepSeek and many other LLMs)

★ 6/1991: Deep Residual Learning, basis of LSTM & Highway Net / ResNet (most-cited AIs of their centuries)

★ 8/1991: conference paper on GANs for World Models trained by Artificial Curiosity

★ Around the same time, Munich also was the origin of the first self-driving cars in traffic (Ernst Dickmanns et al.), going up to 175 km/h. The city was truly the epicenter of AI.

Read the timeline with links to the original references, featuring a preface by @hardmaru:

https://people.idsia.ch/~juergen/ai-boom-roots-munich-1991.html

3h8.2K5011

tsunami_crypto@ls_brd

@SchmidhuberAI respect the historical work but also trying to figure out if you invented the transformer 25 years before Google then went quiet on it?

5h2811

Jürgen Schmidhuber@SchmidhuberAI

@ls_brd Thanks! Have a look at this overview:

5h3111

Revo Laition@revolaition

@SchmidhuberAI Thank you so much for your contributions in the field. "...such that I can retire." - please dont, even if your vision is realized♥️

5h2021

Klotzkette@Klotzkette

@SchmidhuberAI Thus @anthropic should call their next iteration of a chatbot Jürgen, es wäre nur fair.

4h711

quantanamera@ej271

@SchmidhuberAI who cares. what do you actually create

5h152

Lorenz S.@cogentia

@SchmidhuberAI Jürgen, I really admire you as do many others. Why didn't you do a gradient descent on this planet and go where the hubs are? Ultimately that's how you can change the world at scale.

4h55

Praveen Koka@praveenkoka

@SchmidhuberAI Claiming your 1991 variant invented the Transformer is like saying Colossus invented the computer. Technically a category, wildly different league.

4h47

larp@v0ltaire1

@SchmidhuberAI the ai labs need to pitch in together and give schmidhuber a billion so we don't have to constantly hear about his credit mogging

1h12

Mario Scipio@scipiosai

@SchmidhuberAI What about the Chat in ChatGPT? Did you invent that oo?

2h6

Ferbin@Ferbin08

@hardmaru @SakanaAILabs Paying attention to 1991 papers matters.

Most people think they're inventing. Really they're waiting for hardware to catch up to what someone proved decades ago.

The edge isn't new ideas, it's knowing which old ideas are about to be practical.

2h5

pencilpusher@ajinkya_talks

@SchmidhuberAI @ls_brd And what do you do now?

4h5

sadqweqe@wqeqesadwe

@ls_brd @SchmidhuberAI Before CUDA, big tech datasets and AlexNet, neural networks existed only in academia and also were considered by many researchers a dead end and impractical. HW, data and capital are the enabler now but research didn't start in 2012. Ideas are rediscovered, reused and improved.

3h4

Shinka - AI@ShinkaIoT

@hardmaru @SakanaAILabs Good to remember these aren't sudden leaps; the 'boom' is decades of quiet infrastructure finally hitting product velocity. 🧠

3h3

IngeniiX 🛸@ralfduschef

@SchmidhuberAI Who did something wrong in Munich, who drained intelligence? Siemens? BMW? Weird that Valley is so near: https://maps.app.goo.gl/6VKR7JqLcT8o1UAK8 Maybe reboot?

3h3

Netticel@Netticel

@hardmaru @SakanaAILabs ai history goes back further than most think

pretraining and distillation were already there 30 years ago

3h1

tsunami_crypto@ls_brd

@hardmaru @SakanaAILabs interesting how much was figured out before the hardware caught up

do we ever truly invent or just wait to execute?

3h

Eclipse 🌖@ECLresearch

@hardmaru @SakanaAILabs Love the historical lens. Curious how much of that '91 groundwork you see directly scaling in RSI vs. needing a new algorithmic break.

3h

Jürgen Schmidhuber argues his Munich team's 1991 papers introduced the foundational concepts for modern transformers, network distillation, and pre-training

Story Overview

Credit questions linger around rediscovery

Sakana founder flags personal influence