The shortage could force developers to deploy smaller models.
Chris Painter#1472
Are we nearing a compute crunch?
In our latest Gradient Update, @luke__emberson and @Jsevillamol estimate how many tokens all the Blackwell chips on Earth could serve, and compare this to total token demand. Direct comparisons are difficult, but it appears demand is growing much faster than supply.
No Digg Deeper questions have been answered for this story yet.
Another important conclusion on the supply side is that inference is not really compute or bandwidth bound. If you have spare resources, engineers will find ways to use them, using tools like speculative decoding and prefill chunking.
Unfortunately for token demand we have very limited information, so all we can offer are some proxies for growth. I hope we can come back to the topic in a few months with more information and a clearer conceptual framework.
Deep dive into token supply and demand! I come away with the impression that there is going to be significant pressure to keep models intended for the general public small in size.
Are we nearing a compute crunch?
In our latest Gradient Update, @luke__emberson and @Jsevillamol estimate how many tokens all the Blackwell chips on Earth could serve, and compare this to total token demand. Direct comparisons are difficult, but it appears demand is growing much faster than supply.
Our supply estimate is based on serving Kimi K2.6, a trillion-parameter model with 32B active parameters. Using 8k:1k input-to-output token requests, we estimate it would be possible to serve ~20B output tok/s, enough to serve every person on earth 7M tokens per month.
That estimate is highly dependent on the choice of model and inference settings. At longer context lengths, throughput falls substantially. For instance, 128k:1k requests could only be served at around 500M tok/s, a factor of ~50 less.
Our analysis up to here is based only on a rough approximation of a complex process. To calibrate our estimates, we use data from Semianalysis InferenceX, which provides detailed results from real-world inference experiments (including on Kimi 2.5/2.6).
How do our final figures compare to demand? Google just announced that it processes about 1.2B tokens per second (input + output). If we model this volume as 8k:1k requests, that works out to 130M output tokens per second.
Exponential View estimates that Google makes up about 25% of the global demand for tokens. Even if we lavishly insisted on serving all these tokens using expensive trillion-parameter MoE models, that would be enough to serve all current demand.
We build towards this estimate with a simplified model of inference. First, we study prefill and decode, calculating the time for each as a function of batch size. For 8k:1k requests, we find that the prefill phase is compute-bound, while decoding is bandwidth-bound.
But using chunked prefill, we can interleave the prefill and decode phase of different user batches, so that we are only bottlenecked by either compute or memory. We can also use speculative decoding to get multiple output tokens per forward pass with minimal overhead cost.
Available data suggest there are about 14M SWEs using AI daily worldwide. If they matched the intensity of Meta or Apple employees, global throughput demand could reach 200M to 4B tok/s — likely beyond what today's Blackwell fleet could serve at long context lengths.
The Information reported that Meta's 85k employees consumed 60T tokens/month, or ~1M output tokens/day per employee at 25k:1k requests.
Other reports suggest Apple allows some engineering teams to spend $300/day on tokens, enough for 25M Kimi K2.6 output tokens/day per employee.
But growth in supply looks slow relative to trends in demand. Google has experienced a 10×/year increase in tokens processed since 2024. Exponential View finds a similar rate of growth across all providers.
Our simplified analysis misses many factors. Improving capabilities drive demand up, but smaller models also begin to match older, larger ones, displacing some demand. We've also focused on Kimi K2.6; closed frontier models may differ substantially.
The supply of tokens is also steadily increasing. The expansion of AI infrastructure and increases in chip efficiency have expanded global compute by 3.4×/year and memory bandwidth in AI chips by 4.1×/year. Our analysis finds compute growth to be the long-run bottleneck.
@Jsevillamol perhaps an underrated fact that "TAI by 2030" does not just depend on the speed of R&D, but also whether TAI fits in X billion params
probably X is like 10 at most assuming an moe with typical sparsity
Deep dive into token supply and demand! I come away with the impression that there is going to be significant pressure to keep models intended for the general public small in size.
Regardless, our analysis suggests a compute crunch is near, if not already here.
Read the full Gradient Update: https://epoch.ai/gradient-updates/is-a-compute-crunch-coming
@Jsevillamol What do you think about thhe implication of this? Will this lead to power and wealth concentration in the short term? How about the current AI discourse, seems like a lot of people still think AI is a stochastic parrot that can't make anything new
@EpochAIResearch @luke__emberson @Jsevillamol wait this compute vs demand thing feels important
