Reiner Pope builds a multiply-accumulate unit from logic gates and covers systolic arrays, FPGAs, ASICs, and brain architecture in a timestamped blackboard lecture

VIEWS199.6KLIKES1.5K

dwarkesh getting into education is just the best

New blackboard lecture w @reinerpope

How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs

Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

38d199.6K1.5K946

BOOKMARKS1.1KRETWEETS96

Dwarkesh Patel@dwarkesh_sp

"The GPU has a lot of tiny, tiny TPUs tiled across the whole chip." @reinerpope on the architectural difference between the two dominant AI chip designs.

38d68K1.2K1.1K

REPLIES50

Sholto Douglas@_sholtodouglas

Now - starts doing blackboard lectures Next - starts hosting in studio audiences for lectures ... - Dwarkesh university?

Dwarkesh Patel@dwarkesh_sp

New blackboard lecture w @reinerpope

How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs

Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

38d123.4K1.4K369

Dwarkesh Patel@dwarkesh_sp

.@reinerpope's new blackboard lecture goes all the way down: how AI training and inference are built up from logic gates on silicon.

He walks me through a 4-bit multiply-accumulate by hand, and shows how that primitive is the foundation for the matrix multiplies in training runs.

38d31.8K442363

himanshu@himanshustwts

dwarkesh dropping back to back green board bangers.

this time on chip design.

Dwarkesh Patel@dwarkesh_sp

New blackboard lecture w @reinerpope

How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs

Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

38d85.4K744296

Dwarkesh Patel@dwarkesh_sp

Billions of times a second, all the circuitry on an AI chip pauses, just for a moment. Why?

@reinerpope explained to me what's going on, and what the clock cycle of a chip is actually telling us:

38d25.2K250148

Lisan al Gaib@scaling01

Reiner Pope is him

Dwarkesh Patel@dwarkesh_sp

New blackboard lecture w @reinerpope

How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs

Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

38d8.3K5924

Dwarkesh Patel@dwarkesh_sp

Full lecture here:

Dwarkesh Patel@dwarkesh_sp

New blackboard lecture w @reinerpope

How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs

Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

38d11.1K2026

rohit@krishnanrohit

Dwarkesh University

Dwarkesh Patel@dwarkesh_sp

New blackboard lecture w @reinerpope

How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs

Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

38d3.3K2613

Lisan al Gaib@scaling01

@_sholtodouglas he should just create a frontier lab university and make interview and job prep

he would be rich rich

Sholto Douglas@_sholtodouglas

Now - starts doing blackboard lectures Next - starts hosting in studio audiences for lectures ... - Dwarkesh university?

38d1K262

kache@yacineMTB

@dwarkesh_sp @reinerpope thanks for these, really appreciate it

Dwarkesh Patel@dwarkesh_sp

"The GPU has a lot of tiny, tiny TPUs tiled across the whole chip." @reinerpope on the architectural difference between the two dominant AI chip designs.

38d1.4K132

Mahesh Sathiamoorthy@madiator

Dwarkesh RL Environments

Sholto Douglas@_sholtodouglas

Now - starts doing blackboard lectures Next - starts hosting in studio audiences for lectures ... - Dwarkesh university?

38d5.6K82

Tor Tor@tor1457174

@dwarkesh_sp @reinerpope I asked sonnet 3.7 to build a react simulation for systolic array multiply, in like 2 hours, it gave me a fully interactive simulation. That thing is quite interesting, see easy we can observe these algorithm really works today

38d30211

Hayden Sim@hysmio

@dwarkesh_sp @reinerpope getting so spoiled with all these blackboard episodes recently, huge investment in the dwarkesh cinematic universe

38d7628

David Bennett@DavidBennett__

@elliotarledge Especially with @karpathy getting a new job and likely taking a break 😭

38d14711

Yagi@sa_vatsa

The most interesting part the "data tax" and how systolic arrays in TPUs solve it.

Objective of a chip is to do maths but to do that it needs to have data to do the math on.

CPU/GPU works like a personal chef who has to walk to the pantry (registers - data stores), grab a tomato (data), walk back (muxes), chop it, then walk back to the pantry for an onion. Most of the chef's time is spent walking , not chopping.

TPU works like an assembly line (Systolic Array). One worker stays in one spot with their knife (the weight). The tomatoes (data) slide down a conveyor belt. The worker chops as they pass by and slides them to the next person. No one has to walk anywhere, so the "chopping" (math) happens at maximum speed.

By using this "assembly line" design, AI chips can perform quadratically more math while only increasing the "communication" (data movement) by a tiny, linear amount. This is the primary reason modern AI chips are so much faster than general-purpose computers.

38d801

spal@athulspal

@dwarkesh_sp @reinerpope Does DP invest in every company..whose people he interviews ? 👀

38d5242

Danielle Fong 🔆@DanielleFong

when i grow up i'm going to be a graduate at dwarkesh patel university

Dwarkesh Patel@dwarkesh_sp

New blackboard lecture w @reinerpope

How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs

Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

38d1.7K2011

will brown@willccbb

@elliotarledge @_sholtodouglas 👀

38d242

eswarpr@eswarpr

@dwarkesh_sp @reinerpope Watching and listening to this discussion between two smart American technologists, both of whom were born abroad, I wonder sadly about the fate of the U.S. 25-30 years from now as we shut down perfectly legal immigration paths like was announced today.

38d431