Jürgen Schmidhuber argues his Munich team's 1991 papers introduced the foundational concepts for modern transformers, network distillation, and pre-training

VIEWS4.1KBOOKMARKS3LIKES29RETWEETS7REPLIES5

In 1991, the foundations for Transformers, Pre-training, Distillation, and World Models were already being built. These helped shape my own thinking, from my time at Google Brain to our Recursive Self-Improvement (RSI) work at @SakanaAILabs today. 🧠🗼

https://people.idsia.ch/~juergen/ai-boom-roots-munich-1991.html 👇

Jürgen Schmidhuber@SchmidhuberAI

I am proud of the work my team did in Munich in 1991, when compute was millions of times more expensive. We published the roots of today's trillion-dollar AI boom:

★ 3/1991: the first kind of Transformer (see the T in ChatGPT) - now called the unnormalized linear Transformer: the predecessor of the normalized quadratic Transformer

★ 4/1991: Pre-Training (the P in ChatGPT) & Neural Net Distillation (see DeepSeek and many other LLMs)

★ 6/1991: Deep Residual Learning, basis of LSTM & Highway Net / ResNet (most-cited AIs of their centuries)

★ 8/1991: conference paper on GANs for World Models trained by Artificial Curiosity

★ Around the same time, Munich also was the origin of the first self-driving cars in traffic (Ernst Dickmanns et al.), going up to 175 km/h. The city was truly the epicenter of AI.

Read the timeline with links to the original references, featuring a preface by @hardmaru:

https://people.idsia.ch/~juergen/ai-boom-roots-munich-1991.html

1h4.1K293

tsunami_crypto@ls_brd

@SchmidhuberAI respect the historical work but also trying to figure out if you invented the transformer 25 years before Google then went quiet on it?

3h2811

Jürgen Schmidhuber@SchmidhuberAI

@ls_brd Thanks! Have a look at this overview:

3h3111

Revo Laition@revolaition

@SchmidhuberAI Thank you so much for your contributions in the field. "...such that I can retire." - please dont, even if your vision is realized♥️

3h2021

Klotzkette@Klotzkette

@SchmidhuberAI Thus @anthropic should call their next iteration of a chatbot Jürgen, es wäre nur fair.

1h711

quantanamera@ej271

@SchmidhuberAI who cares. what do you actually create

3h152

Lorenz S.@cogentia

@SchmidhuberAI Jürgen, I really admire you as do many others. Why didn't you do a gradient descent on this planet and go where the hubs are? Ultimately that's how you can change the world at scale.

1h55

Praveen Koka@praveenkoka

@SchmidhuberAI Claiming your 1991 variant invented the Transformer is like saying Colossus invented the computer. Technically a category, wildly different league.

1h47

Ferbin@Ferbin08

@hardmaru @SakanaAILabs Paying attention to 1991 papers matters.

Most people think they're inventing. Really they're waiting for hardware to catch up to what someone proved decades ago.

The edge isn't new ideas, it's knowing which old ideas are about to be practical.

22m5

pencilpusher@ajinkya_talks

@SchmidhuberAI @ls_brd And what do you do now?

2h5

sadqweqe@wqeqesadwe

@ls_brd @SchmidhuberAI Before CUDA, big tech datasets and AlexNet, neural networks existed only in academia and also were considered by many researchers a dead end and impractical. HW, data and capital are the enabler now but research didn't start in 2012. Ideas are rediscovered, reused and improved.

1h4

Shinka - AI@ShinkaIoT

@hardmaru @SakanaAILabs Good to remember these aren't sudden leaps; the 'boom' is decades of quiet infrastructure finally hitting product velocity. 🧠

42m3

IngeniiX 🛸@ralfduschef

@SchmidhuberAI Who did something wrong in Munich, who drained intelligence? Siemens? BMW? Weird that Valley is so near: https://maps.app.goo.gl/6VKR7JqLcT8o1UAK8 Maybe reboot?

1h3

Netticel@Netticel

@hardmaru @SakanaAILabs ai history goes back further than most think

pretraining and distillation were already there 30 years ago

1h1

tsunami_crypto@ls_brd

@hardmaru @SakanaAILabs interesting how much was figured out before the hardware caught up

do we ever truly invent or just wait to execute?

1h

Eclipse 🌖@ECLresearch

@hardmaru @SakanaAILabs Love the historical lens. Curious how much of that '91 groundwork you see directly scaling in RSI vs. needing a new algorithmic break.

1h

Jürgen Schmidhuber argues his Munich team's 1991 papers introduced the foundational concepts for modern transformers, network distillation, and pre-training

Story Overview

Credit questions linger around rediscovery

Sakana founder flags personal influence