New Hardware Drops Inference Costs for Massive MoE Models
Reactions from ranked influencers
2 posts
Token economics are so wild these days. The prices for models are basically pinned to when they come out as a function of what their new hardware is. As a result if you don’t want to do a custom deployment it’s often cheaper to use a hundreds of billion parameter MoE than a single digit dense model or even an MoE in the tens of parameters from 6-8 months ago.
There is probably an economies-of-scale aspect to it, too. All the investment is going into accelerating a certain class of model. So even if a model has fewer FLOPs, it might just be a bad fit for the new hardware or just not optimized.
Token economics are so wild these days. The prices for models are basically pinned to when they come out as a function of what their new hardware is. As a result if you don’t want to do a custom deployment it’s often cheaper to use a hundreds of billion parameter MoE than a single digit dense model or even an MoE in the tens of parameters from 6-8 months ago.