Developer Runs 29-Million-Parameter Model On $8 ESP32 Microcontroller
Reactions from ranked influencers
2 posts
WOW! The $8 AI Machine! Something extraordinary just happened and it changes what “local AI” can mean. I am testing it tonight. Thus far it shows many possibilities… So what it this $8 AI device? A developer going by slvDev has forced a 28.9-million-parameter language model onto an ESP32-S3 microcontroller that costs roughly eight dollars. Not a Raspberry Pi. Not a Jetson. An eight-dollar microcontroller. The model runs completely offline, generates coherent short stories at about 9.5 tokens per second, and draws power measured in the same range as a small LED. This is more than a hundred times larger than the previous record for the same class of chip (the earlier 260,000-parameter TinyStories experiments). For perspective, the original ChatGPT sat at 117 million parameters. We are now running a model roughly a quarter of that size on silicon you can buy for the price of two coffees. How the Impossible Became Possible The ESP32-S3 has only 512 KB of fast SRAM, 8 MB of PSRAM, and 16 MB of flash. Conventional wisdom said a model of this size simply would not fit. The breakthrough is architectural, not brute force. Most of a language model’s parameters live in a giant embedding table a lookup table you mostly read from, not compute against. Drawing directly from Google’s Per-Layer Embeddings technique (the same family of ideas used in the Gemma models), the developer moved the bulk of that table roughly 25 million parameters into flash memory and memory-mapped it. The chip only needs to pull about six rows, roughly 450 bytes, for each new token. The remaining dense “thinking” core stays in the fast SRAM (around 560 K of active working memory). The model is stored at 4-bit quantization and occupies about 14.9 MB total. The result is a system that feels almost free to run. The heavy parameters sit quietly in flash and are sampled sparingly. The little core does the real work. It is elegant engineering of the purest kind. What I Am Doing With It Right Now I have the boards on the bench in the garage lab. The first units are already talking short, coherent stories appearing on a tiny wired display, generated entirely on the chip with no Wi-Fi, no API key, no cloud round-trip. Latency is local. Privacy is absolute. Power draw is low enough that battery operation becomes interesting. I am treating these as the first generation of true $8 AI machines. Early tests are focused on three practical directions. - Embedding the model into simple nodes. - Pairing it with local voice front-ends - Exploring whether multiple of these chips can be networked as a lightweight swarm. The model is deliberately limited. It was trained on the Microsoft TinyStories dataset and is excellent at coherent narrative, not at open-ended question answering or tool use. That is a feature, not a bug. It forces us to design systems around what the silicon can actually deliver instead of pretending every edge device needs a frontier model. Real Use Cases That Suddenly Become Practical Once you accept that a capable language model can live for eight dollars and run without the cloud, a new class of devices becomes possible: This is the opposite of the current trajectory that wants every intelligent act to travel through a remote server. It is the beginning of intelligence that is cheap enough, private enough, and local enough to become infrastructure rather than a service. We have spent years watching model sizes explode upward. The more interesting frontier may be the opposite direction: how small, how cheap, and how local can useful intelligence become? An eight-dollar chip that can tell coherent stories is not a toy. It is a proof that the lower bound keeps moving. The open repository is at https://github.com/slvDev/esp32-ai I will keep testing, measuring, and reporting what these little machines can and cannot do. The age of abundant local intelligence just got a little more real, and it arrived wearing an eight-dollar price tag.
THIS IS AN ENTIRE SELF CONTAINED $8 AI COMPUTER! ESP32-AI adds a 28.9M LLM to a microcontroller. It can store 28.9 million parameters; for context, the very first version of ChatGPT had 117 million parameters. Thus, this ESP32 project only has about a quarter of the smarts of OpenAI's first model, but given how this thing was squashed onto an ESP32 device, that's a far more impressive feat than it might sound at first. This opens up a massive opportunity for dozens of these devices in a network working on specific domains and returning results to a master ESP32. This is my plan…
WOW! The $8 AI Machine! Something extraordinary just happened and it changes what “local AI” can mean. I am testing it tonight. Thus far it shows many possibilities… So what it this $8 AI device? A developer going by slvDev has forced a 28.9-million-parameter language model onto an ESP32-S3 microcontroller that costs roughly eight dollars. Not a Raspberry Pi. Not a Jetson. An eight-dollar microcontroller. The model runs completely offline, generates coherent short stories at about 9.5 tokens per second, and draws power measured in the same range as a small LED. This is more than a hundred times larger than the previous record for the same class of chip (the earlier 260,000-parameter TinyStories experiments). For perspective, the original ChatGPT sat at 117 million parameters. We are now running a model roughly a quarter of that size on silicon you can buy for the price of two coffees. How the Impossible Became Possible The ESP32-S3 has only 512 KB of fast SRAM, 8 MB of PSRAM, and 16 MB of flash. Conventional wisdom said a model of this size simply would not fit. The breakthrough is architectural, not brute force. Most of a language model’s parameters live in a giant embedding table a lookup table you mostly read from, not compute against. Drawing directly from Google’s Per-Layer Embeddings technique (the same family of ideas used in the Gemma models), the developer moved the bulk of that table roughly 25 million parameters into flash memory and memory-mapped it. The chip only needs to pull about six rows, roughly 450 bytes, for each new token. The remaining dense “thinking” core stays in the fast SRAM (around 560 K of active working memory). The model is stored at 4-bit quantization and occupies about 14.9 MB total. The result is a system that feels almost free to run. The heavy parameters sit quietly in flash and are sampled sparingly. The little core does the real work. It is elegant engineering of the purest kind. What I Am Doing With It Right Now I have the boards on the bench in the garage lab. The first units are already talking short, coherent stories appearing on a tiny wired display, generated entirely on the chip with no Wi-Fi, no API key, no cloud round-trip. Latency is local. Privacy is absolute. Power draw is low enough that battery operation becomes interesting. I am treating these as the first generation of true $8 AI machines. Early tests are focused on three practical directions. - Embedding the model into simple nodes. - Pairing it with local voice front-ends - Exploring whether multiple of these chips can be networked as a lightweight swarm. The model is deliberately limited. It was trained on the Microsoft TinyStories dataset and is excellent at coherent narrative, not at open-ended question answering or tool use. That is a feature, not a bug. It forces us to design systems around what the silicon can actually deliver instead of pretending every edge device needs a frontier model. Real Use Cases That Suddenly Become Practical Once you accept that a capable language model can live for eight dollars and run without the cloud, a new class of devices becomes possible: This is the opposite of the current trajectory that wants every intelligent act to travel through a remote server. It is the beginning of intelligence that is cheap enough, private enough, and local enough to become infrastructure rather than a service. We have spent years watching model sizes explode upward. The more interesting frontier may be the opposite direction: how small, how cheap, and how local can useful intelligence become? An eight-dollar chip that can tell coherent stories is not a toy. It is a proof that the lower bound keeps moving. The open repository is at https://github.com/slvDev/esp32-ai I will keep testing, measuring, and reporting what these little machines can and cannot do. The age of abundant local intelligence just got a little more real, and it arrived wearing an eight-dollar price tag.