Reiner Pope builds a multiply-accumulate unit from logic gates and covers systolic arrays, FPGAs, ASICs, and brain architecture in a timestamped blackboard lecture
Dwarkesh Patel shared the lecture spanning 0:00:00 to 1:11:49.
.@reinerpope's new blackboard lecture goes all the way down: how AI training and inference are built up from logic gates on silicon.
He walks me through a 4-bit multiply-accumulate by hand, and shows how that primitive is the foundation for the matrix multiplies in training runs.
#70Dwarkesh Patel@DWARKESH_SPFull lecture here:
Dwarkesh Patel@dwarkesh_spNew blackboard lecture w @reinerpope How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do. 0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!
Billions of times a second, all the circuitry on an AI chip pauses, just for a moment. Why?
@reinerpope explained to me what's going on, and what the clock cycle of a chip is actually telling us:
#94Sholto Douglas@_SHOLTODOUGLASNow - starts doing blackboard lectures Next - starts hosting in studio audiences for lectures ... - Dwarkesh university?
Dwarkesh Patel@dwarkesh_spNew blackboard lecture w @reinerpope How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do. 0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!
@_sholtodouglas he should just create a frontier lab university and make interview and job prep
he would be rich rich
Sholto Douglas@_sholtodouglasNow - starts doing blackboard lectures Next - starts hosting in studio audiences for lectures ... - Dwarkesh university?
#980Lisan al Gaib@SCALING01Reiner Pope is him
Dwarkesh Patel@dwarkesh_spNew blackboard lecture w @reinerpope How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do. 0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!
#1220rohit@KRISHNANROHITDwarkesh University
Dwarkesh Patel@dwarkesh_spNew blackboard lecture w @reinerpope How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do. 0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!