/Tech1h ago

Vision Transformer co-author Lucas Beyer says the LLM community is rediscovering 2019 computer vision insights on weight decay and training epochs

Story Overview

Lucas Beyer points out that today's frontier labs are hitting the same levers his 2019 S4L work identified on ImageNet subsets: crank weight decay while extending training on reused data, and performance keeps climbing even after the token count reaches hundreds of trillions. The old experiments showed architecture depth and width mattered far less than those two knobs, a result now being echoed in LLM pre-training runs that pair high weight decay with distillation and synthetic data.

19234912719.4K

#72

Original post

Lucas Beyer (bl16)@giffmana#72inTech

LLM community slowly rediscovering what we in vision found out over half a decade ago. MY SCHMIDHUBER MOMENT IS COMING!

Source: S4L paper where i tuned the most sota 10% and 1% ImageNet baselines ever, by far. https://arxiv.org/abs/1905.03670

Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTex

for people wondering how frontier labs can scale to hundreds of trillions of tokens: just crank weight decay ALL THE WAY UP and keep grinding on the same dataset, silly! Lots of other details on distillation, ensembling, synthetic data too No, tokens won't be a wall

1:21 AM · Jun 15, 2026 · 15.7K Views

Research Echo

What the original ablations actually measured

Thousands of ResNet variants were trained on 1 % and 10 % ImageNet labels; the clearest signal was that weight-decay values around 10^{-3} or 10^{-2} combined with 200-to-1000-epoch schedules dominated final accuracy, while swapping ResNet-18 for ResNet-50 or doubling width produced only marginal shifts.

Open Question

How far the analogy actually stretches

No side-by-side numbers yet map the 2019 ImageNet regimes to current LLM token budgets or confirm identical weight-decay magnitudes, so the parallel remains observational rather than quantified.

Sentiment

Positive users are excited about the hyperparameter scaling insights and nGPT normalized transformers paper due to advantages for GPU-poor researchers, while negative users see the claims as mere arbitrage from vision research.

Pos

90.0%

Neg

10.0%

8 comments with sentiment.

Cluster Engagement

Posts from X

Most Activity

VIEWS2.8KBOOKMARKS13LIKES32REPLIES3

Susan Zhang@suchenzang

coming in hot to rain on lucas's schmidhuber poasting 🥸

Lucas Beyer (bl16)@giffmana

LLM community slowly rediscovering what we in vision found out over half a decade ago. MY SCHMIDHUBER MOMENT IS COMING!

Source: S4L paper where i tuned the most sota 10% and 1% ImageNet baselines ever, by far. https://arxiv.org/abs/1905.03670

53m2.8K3213

RETWEETS2

Lucas Beyer (bl16)@giffmana

@suchenzang Little did she know, we already weight-normalized back in BiT in 2019, too.

You can't out-Schmidhuber me, Susan.

Susan Zhang@suchenzang

coming in hot to rain on lucas's schmidhuber poasting 🥸

49m602130

Lucas Beyer (bl16)@giffmana

@suchenzang Hehe I knew you would find and say that!

wd still had a pretty strong effect as we did wd the raw buffers but do norm as part of the graph, which i believe is different than nGPT. Plus there's always a few non-normed params.

Susan Zhang@suchenzang

@giffmana and kept weight decay?? absolute madlads

34m17022

stochasm@stochasticchasm

@suchenzang i'm a fan of this one https://arxiv.org/abs/2603.28743v2

49m11122

Susan Zhang@suchenzang

@giffmana and kept weight decay?? absolute madlads

Lucas Beyer (bl16)@giffmana

@suchenzang Little did she know, we already weight-normalized back in BiT in 2019, too.

You can't out-Schmidhuber me, Susan.

45m49161

Susan Zhang@suchenzang

@giffmana i am forever a stan of your hyperparams :)

Susan Zhang@suchenzang

All the weight decay anecdotes in [15,46,22] smells a bit like there's a learning rate confounder in the mix somewhere...

Wondering where weight decay gets applied in all of these experiments (@giffmana is https://arxiv.org/abs/1711.05101 the default?).

Delightful read, as always!

4/4

23m12622

Qianyi Zhang@QianyiZhan95891

@giffmana vision guru should respond with vision language

42m1811

Lucas Beyer (bl16)@giffmana

Note: the LLM paper this is about had more, like ensembles and distilling them, but... I'll leave that for another day lol

Lucas Beyer (bl16)@giffmana

LLM community slowly rediscovering what we in vision found out over half a decade ago. MY SCHMIDHUBER MOMENT IS COMING!

Source: S4L paper where i tuned the most sota 10% and 1% ImageNet baselines ever, by far. https://arxiv.org/abs/1905.03670

1h1.5K80

tuōmo@7uomoki

@giffmana you have to make a more in-your-face post/thread like "Who originally invented the universal laws of pre-training under infinite compute and optimal weight decay?" the answer is of course you, but then the thread is also a list of 20 other articles that were not cited! 😉

1h1842

Lucas Beyer (bl16)@giffmana

@7uomoki Hahaha yeah i still have about 20 years left to finesse this skill

1h1454

Ethan@torchcompiled

@giffmana "depth, and width only have a small influence on the final results"

Big moment for gpu-poor guys like me in taking my GPT2-350M to go up against Fable5

1h883

Lucas Beyer (bl16)@giffmana

@QianyiZhan95891 Hahaha those are absolutely beautiful thank you!!

33m7

M Saiful Bari (MARUF)@sbmaruf

@giffmana Weight decay is a critical thing to model but with a proper ladder decoupling specific hps.

1h1421

Yitong chen@1997yrrr

@giffmana 🤣 I know this since your lastest talk

1h1281

Hynek Kydlíček@HKydlicek

@giffmana epochs is completely artificial concept 🫠

1h611

Tim@daidailoh

@giffmana lel I didn't even know that they were not already cranking up the WD... guess they were adjusting WD with just a smaller model instead of also reducing the training tokens when they swept for hyper params?

computer vision people superiority proof right there :D

39m211

Ignacio de Gregorio@thewhiteboxAI

@giffmana How related is this to grokking? Is it something that is done in modern training? Just regularizing to the maximum?

54m64

Patrick Baitman@taofanqq

@giffmana That's the arbitrage, find some b tier lab and port what has already been done in vision as novel llm research

1h55

Plugyawn@plugyawn

@giffmana I'm again a greenhorn but would you say this is correct: as long as the learning system is the same (backprop over nonlinear nets), the architecture can be abstracted out as "how many learning steps per second on the target hardware"? I have so many questions! I wish I could DM.

20m47

Lazarz@Laz4rz

@stochasticchasm @suchenzang Nerdsnipe

38m15