/Tech2d ago

Dwarkesh Podcast host Dwarkesh Patel argues AI's million-fold human sample efficiency gap makes data volume the primary driver of gains

This allows open-source developers to quickly replicate frontier models

535633430577.9K
judah
shyamal
Adam Marblestone
Nathan Lambert
Dwarkesh Patel#67
Original post
Dwarkesh Patel@dwarkesh_sp#67inTech

New blog post: on the million-x sample efficiency gap between AIs and humans, and whether it matters:

"The reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress.

And data can be easily distilled from public APIs, whereas hyper-parameters and training tricks and architectural micro-optimizations cannot - if the latter were driving most of progress, then catching up would be harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes.

We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data."

Post in link below

11:10 AM · Jun 8, 2026 · 131.2K Views
Sentiment

Positive users praise articles on data's underrated scale in training AI models and enabling open-source catch-up, while negative users criticize reliance on underpaid labelers and dismiss related concepts like distillation as cope.

Pos
57.1%
Neg
42.9%
9 comments with sentiment.
Cluster Engagement
Posts from X
Most Activity
Most Activity
VIEWS68.9KBOOKMARKS276LIKES473RETWEETS32REPLIES39
Nathan Lambert@natolambert

I feel like the obsession with continual learning / sample efficiency leads the field in the wrong direction. It's the bad career strategy of focusing on addressing your weaknesses instead of maximizing your strengths.

Yes, there is an existence proof in the human brain, but it doesn't by any means guarantee that that'll be the most interesting AI. It may require $100T of R&D on chips and AI methods to get that unlock.

On the other side of things, it's obvious that the coming models are extremely transformative and built on technologies that we already have. There's great reason to focus on just maximizing this. In reality, this is what the frontier labs are doing. They're going as fast as possible down the current development tree. This is good for progress and mixed for safety/geopolitics.

Things like "automate white color work" and "replace the AI researcher job" are the guesses of labs because it's super hard to imagine futures for what these dramatic technologies will be. Don't take the labs too seriously about this being the exact goal. The exact goal is to push the frontier and monetize later.

Solving continual learning, sample efficiency, etc would be great, but its trying to predict when a scientific breakthrough will come instead of trying to grapple with how the 100% sure thing coming technological revolution will change our lives.

This isn't to say the Dwarkesh post is bad, it addresses some reasonable critiques, but it is the least bitter lesson pilled thing to be obsessed with human intelligence and how that can inform AI.

We are in the AGI era of research. This is about embracing the unknown, scaling resources, and seeing what is enabled by making a series of magical tweaks to complex recipes that build frontier models. Lean into the alchemy.

(it should be pretty clear that I personally, investing in open research agree we need fundamental science -- just not agreeing that this is what the "cutting edge of the frontier" is governed by)

Dwarkesh Patel@dwarkesh_sp

New blog post: on the million-x sample efficiency gap between AIs and humans, and whether it matters:

"The reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress.

And data can be easily distilled from public APIs, whereas hyper-parameters and training tricks and architectural micro-optimizations cannot - if the latter were driving most of progress, then catching up would be harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes.

We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data."

Post in link below

1dViews 68.9KLikes 473Bookmarks 276
Dwarkesh Patel@dwarkesh_sp

https://www.dwarkesh.com/p/the-sample-efficiency-black-hole

Dwarkesh Patel@dwarkesh_sp

New blog post: on the million-x sample efficiency gap between AIs and humans, and whether it matters:

"The reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress.

And data can be easily distilled from public APIs, whereas hyper-parameters and training tricks and architectural micro-optimizations cannot - if the latter were driving most of progress, then catching up would be harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes.

We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data."

Post in link below

2dViews 12.8KLikes 85Bookmarks 84
judah@joodalooped

all aboard the data train!

https://anjalishriva.com/work-data/

Dwarkesh Patel@dwarkesh_sp

New blog post: on the million-x sample efficiency gap between AIs and humans, and whether it matters:

"The reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress.

And data can be easily distilled from public APIs, whereas hyper-parameters and training tricks and architectural micro-optimizations cannot - if the latter were driving most of progress, then catching up would be harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes.

We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data."

Post in link below

2dViews 9.5KLikes 59Bookmarks 46
Nathan Lambert@natolambert

the crux of my ick is the link to the human brain. Just saying the products aren't good enough is fine.

Nathan Lambert@natolambert

I feel like the obsession with continual learning / sample efficiency leads the field in the wrong direction. It's the bad career strategy of focusing on addressing your weaknesses instead of maximizing your strengths.

Yes, there is an existence proof in the human brain, but it doesn't by any means guarantee that that'll be the most interesting AI. It may require $100T of R&D on chips and AI methods to get that unlock.

On the other side of things, it's obvious that the coming models are extremely transformative and built on technologies that we already have. There's great reason to focus on just maximizing this. In reality, this is what the frontier labs are doing. They're going as fast as possible down the current development tree. This is good for progress and mixed for safety/geopolitics.

Things like "automate white color work" and "replace the AI researcher job" are the guesses of labs because it's super hard to imagine futures for what these dramatic technologies will be. Don't take the labs too seriously about this being the exact goal. The exact goal is to push the frontier and monetize later.

Solving continual learning, sample efficiency, etc would be great, but its trying to predict when a scientific breakthrough will come instead of trying to grapple with how the 100% sure thing coming technological revolution will change our lives.

This isn't to say the Dwarkesh post is bad, it addresses some reasonable critiques, but it is the least bitter lesson pilled thing to be obsessed with human intelligence and how that can inform AI.

We are in the AGI era of research. This is about embracing the unknown, scaling resources, and seeing what is enabled by making a series of magical tweaks to complex recipes that build frontier models. Lean into the alchemy.

(it should be pretty clear that I personally, investing in open research agree we need fundamental science -- just not agreeing that this is what the "cutting edge of the frontier" is governed by)

1dViews 4.8KLikes 50Bookmarks 3
judah@joodalooped

few understand this!

2dViews 486Likes 9Bookmarks 2
Adam Marblestone@AdamMarblestone

"The labs’ plan for these later kinds of jobs is to first automate AI research, and then have the automated AI researchers solve this sample efficiency problem. So then the question is, can AIs, which do not have human-level sample efficiency, nonetheless solve the remaining research problems on the way to human-like intelligence and learning." Right! Are there no skills involved in AI research that are either not elastic relative to Type 1 or Type 2 trainable labor, or are Type 3 skills? https://longitudinal.blog/2023/01/10/general-automation-and-science/ RSI's asymptote obviously depends on its initial condition?

"Scaling laws tell us that bigger models are more sample efficient. The human brain is 100T synapses - if each synapse is ~1 parameter, and frontier models are currently roughly ~5T parameters, then maybe we could achieve human-level sample efficiency with another order of magnitude or two of parameter scaling." If I understood this @gwern blog post correclty I think he is saying to abandon Chinchilla scaling law per se but still use a massively over-parametrized neural net? https://gwern.net/llm-catapult

Dwarkesh Patel@dwarkesh_sp

https://www.dwarkesh.com/p/the-sample-efficiency-black-hole

1dViews 612Likes 1Bookmarks 3
Aravind Jayendran@Maxaravind

Here is a thought: LLMs are super sample efficient at learning things in context. It's just that ICL is limited as of now. What we need are mechanisms to somehow absorb the information stored "in-context" into the weights. This is in a sense what fast weights do in GDNs to an extent.

Then the question becomes how do we train LLMs to learn to efficiently pack information into fast weights( and eventually slow weights ) over very long horizons. This is a problem that can be solved if trained on the right reward functions and tasks imo. Maybe just a matter of time.

In that case, we could still think of pre training as rolling up a base arch, similar to how evolution produced the human, brain with the difference that our pre training and post training still has not produced an arch that can do long range in-contex -> fast weights -> slow weights effectively.

Wrote about bit about it here: https://www.aravindjayendran.com/writing/few-shot-learners-cant-remember

2dViews 186Likes 2Bookmarks 3
Jacob Portes@JacobianNeuro

@dwarkesh_sp I find motor control to be a particularly compelling example of the discrepancy between human and AI sample efficiency:

1dViews 237Likes 1Bookmarks 2
Nathan Lambert@natolambert

@___Harald___ I agree -- but the labs are focused on the most useful technology quickly, else they financially implode

1dViews 281Likes 6Bookmarks 1
shyamal@shyamalanadkat

more human data is not the path to agi

Dwarkesh Patel@dwarkesh_sp

New blog post: on the million-x sample efficiency gap between AIs and humans, and whether it matters:

"The reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress.

And data can be easily distilled from public APIs, whereas hyper-parameters and training tricks and architectural micro-optimizations cannot - if the latter were driving most of progress, then catching up would be harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes.

We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data."

Post in link below

1dViews 504Likes 4Bookmarks 0
dj microbeads@djmicrobeads

@joodalooped excellent article

2dViews 148Likes 3
Asuka Zheng🎀@VoidAsuka

@dwarkesh_sp good writing, thank u! i like this take - Many billions of years of evolution is our pre-training, so it’s unfair to compare how little data we see simply within our lifetime to what these cold-started LLMs have to learn from.

1dViews 378Likes 6
Harald Schäfer@___Harald___

@natolambert I would like to see a lot more open research inspired by biological intelligence. Specifically on things like sample efficiency, sparse/delayed rewards, and digital evolution.

They may not lead to the most useful technology most quickly, but it can answer interesting questions.

1dViews 182Likes 1
Jeffcafe, private detective@jeffcafe_

@dwarkesh_sp Counterpoint: techniques aren’t a bottleneck because knowledge diffuses very easily due to distillation, publications, staff movements, and open sourcing. Consider that Llama 3 open sourcing brought everyone to frontier level overnight.

2dViews 434Likes 3
Ilman Shazhaev@shzhv13

@natolambert Interesting how the frontier progress and product progress aren’t the same game. Labs can optimize for the next jump, founders have to optimize for usable systems in the present.

1dViews 72Likes 1
Lorah@seekingyaga

@dwarkesh_sp To riff on the black hole analogy, we can see the pull of data in LLM outputs especially when using the same prompt in different models. The recent "what character am I" meme really illustrated the differences between models.

1dViews 69
Aria Westcott@AriaWestcott

@dwarkesh_sp This post actually says something, which is rare.

Most of the cycle is just vibes with a thread on top.

1dViews 2.1KLikes 1
JJ@JosephJacks_

@natolambert 😬

1dViews 675Likes 1
magica 📜@jaini4mtheblock

@natolambert @___Harald___ I think it is easier for non-ML people to be drawn by "how humans work" (me included) because of a lack of reference points, whereas ML people may have a lot better mechanical intuition (rejig the kv cache and you'll get a lot more foo with a lot less dimensions or something)

1dViews 20
Load more posts