it is truly amazing we can supervise a trillion parameters from 16 bits of information. the fact that credit assignment across such a massive & discontinuous network is such a non-issue deserves so much reflection
5:21 PM · Jun 8, 2026 · 7.4K Views
OpenAI's Will Depue says trillion-parameter models can be supervised using only 16 bits of information
Charles Foster says batch and sequence dimensions facilitate this.
81202167.7K
Sentiment
Positive users note catastrophic forgetting is nearly a non-issue at scale for trillion-parameter models, while negative users dismiss 16-bit supervision as still excessive.
Pos
50.0%
Neg
50.0%
2 comments with sentiment.
Cluster Engagement
Posts from X
Most Activity
Most Activity
VIEWS86
Henry Dowling@henrytdowling
@willdepue why do you think that is? high dimensions just dont really have that many local minima so it will "eventually work"?
5hViews 86
LIKES3REPLIES1
Charles Foster@CFGeek
@willdepue @lu_sichu *batch/sequence dimension voice*
4hViews 48Likes 3
will depue@willdepue
@CFGeek @lu_sichu lol
4hViews 48Likes 1
Santosh Mohan@theycallmeMohan
@willdepue Catastrophic forgetting almost a non-issue as well at scale!
5hViews 52
Cliff Lattner@CliffLattner
@willdepue 16 bits is a metric fuckton
4hViews 28