Reactions from ranked influencers
2 posts
As someone who's been shipping LLMs since the GPT-2 days, this lecture on cross-entropy from a Stanford math grad is the closest thing to an ML PhD qualifying exam I've ever seen released publicly for free. Everyone thinks language models predict the next word. They don't. They compress language. Once you see the math, you can't unsee it. 33 minutes. Bookmark & watch today.
Prediction is compression. Same thing.
As someone who's been shipping LLMs since the GPT-2 days, this lecture on cross-entropy from a Stanford math grad is the closest thing to an ML PhD qualifying exam I've ever seen released publicly for free. Everyone thinks language models predict the next word. They don't. They compress language. Once you see the math, you can't unsee it. 33 minutes. Bookmark & watch today.