Engineer Johnny Lee says Together AI's monthly volume of 400 trillion tokens puts its scale one year behind Google

So @togethercompute is now ~1year behind Google scale, which is crazy

So they are now ~1year behind Google scale, which is crazy

@natolambert 400T/month is a wild number. Feels like tokens have become the new unit for watching AI infrastructure scale in real time.

@natolambert That’s only 50k GPUs… much smaller than I imagined

@natolambert That was the only useful info in this very long boring article

@natolambert is that a run rate number or did they actually cross it already?

curious how much is inference vs training

@natolambert 13T+ a day. Big numbers!

@natolambert 400T tokens/month really puts inference-time compute into perspective. That's more throughput than most people imagine when they think about "training vs. serving" economics.

@natolambert 400T token etkileyici, evet. ama çoğu zaman bu hız, inovasyonun önüne geçip yalnızca mevcut mimarileri daha verimli kullanma çabasına dönüşüyor. yeni bir şey mi üretiyoruz, yoksa sadece daha hızlı mı koşuyoruz?

@natolambert And here I'm, thinking near 10T per month is wild.

@natolambert 400T tokens a month is the kind of number that stops sounding digital.

At that scale, small UI defaults become infrastructure. One bad retry button or hidden cache decision turns into an enormous invisible bill.

@ByJohnnyLee @togethercompute exponentials doing a lot of work for this comparison though

@natolambert