Pierre-Carl Langlais highlights how persistent equipment delays keep many GPU setups stuck at modest sizes, directly complicating cost-effective inference for large Mixture-of-Experts models that benefit from bigger batch runs.
@willccbb Yeah that was the point I intended to develop in next post. As equipment lags, many cos have GPUs lying around but more in the 16-32 h100 range.
@Dorialexander as with all big MoEs, economical serving kicks in at fairly high batch size
32 or 64 gpus is common
Cost-effective MoE serving often calls for 32-64 GPU batches, yet supply lags leave many operators with only half that scale and force trade-offs in throughput or economics.
Participants note rising interest in organizations that can aggregate or secure larger clusters at better prices, though no specific providers or pricing details have surfaced yet.
No Digg Deeper questions have been answered for this story yet.
@Dorialexander @willccbb I wonder whether there’s an org which helps you sourcing larger clusters for your needs at great prices
@willccbb Yeah that was the point I intended to develop in next post. As equipment lags, many cos have GPUs lying around but more in the 16-32 h100 range.
@Dorialexander guessing we get something close to glm-5.2-ish in the 400-500b range in the next few months? our results there were h200 fp8, should translate to something a bit smaller
tuning is fiddly but doable