EngramLab's Dan Biderman and Jessy Lin propose shifting the context layer into model weights to enable continuous learning

Today's AI models train once. We don't work that way. We learn continuously, forget what doesn't matter, and retain what does.

That gap is what @dan_biderman and @realJessyLin are closing at @EngramLab. AI that never stops learning, with memory that lives inside the model instead of bolted on as an afterthought.

In our latest Training Data episode we get into why memory is the next frontier: why the brain forgets on purpose, why RAG is a band-aid, and what becomes possible when a model is always training.

00:00 Introduction 00:59 Always Training Explained 01:51 Beyond Context Windows 03:29 Ngram Product Overview 04:34 Adapters And Training Signals 05:32 Internalize Vs Externalize 06:49 Compute And Token Savings 08:19 Teams First Then Individuals 08:51 Memorization Vs Understanding 12:47 Dreams And Offline Digestion 14:08 Training Beats Curation 15:19 Why Everyone Needs A Model 21:44 Bitter Lesson And Architecture 24:44 RAG Killer And KV Cache 31:38 Future Of Memory And Models

Today's AI models train once. We don't work that way. We learn continuously, forget what doesn't matter, and retain what does.

That gap is what @dan_biderman and @realJessyLin are closing at @EngramLab. AI that never stops learning, with memory that lives inside the model instead of bolted on as an afterthought.

In our latest Training Data episode we get into why memory is the next frontier: why the brain forgets on purpose, why RAG is a band-aid, and what becomes possible when a model is always training.

00:00 Introduction 00:59 Always Training Explained 01:51 Beyond Context Windows 03:29 Ngram Product Overview 04:34 Adapters And Training Signals 05:32 Internalize Vs Externalize 06:49 Compute And Token Savings 08:19 Teams First Then Individuals 08:51 Memorization Vs Understanding 12:47 Dreams And Offline Digestion 14:08 Training Beats Curation 15:19 Why Everyone Needs A Model 21:44 Bitter Lesson And Architecture 24:44 RAG Killer And KV Cache 31:38 Future Of Memory And Models