OpenAI quietly found an optimization that cuts inference costs in half on the models it touched, and the first visible payoff showed up in logged-out ChatGPT traffic where GPU count dropped to just a few hundred. The work came from engineers squeezing more out of existing servers rather than chasing new chips, a detail that stayed internal until The Information surfaced it.
OpenAI has found a way to cut inference costs in half.
OpenAI engineers earlier this month developed an optimization that cut inference costs in half for models it was applied to.
After the optimization was applied to logged-out ChatGPT traffic, it reduced the number of GPUs needed to power that traffic to a couple hundred.
The change only targeted logged-out traffic so far, leaving open how much headroom remains for the rest of the service or other products.
Anthropic and Google are chasing the same server-level gains, yet no public benchmarks or code have appeared, so the exact trick and its broader applicability stay unknown for now.
Positive users call OpenAI's optimizations that halve ChatGPT inference costs a major efficiency leap, while negative users blame the changes for recent drops in model quality.
No Digg Deeper questions have been answered for this story yet.
#1360OpenAI reportedly found new inference optimizations that more than halved the cost of running its models!
According to The Information, engineers told colleagues this month that the techniques helped power ChatGPT for visitors without free or paid accounts using only a couple hundred Nvidia GPUs at one point.
The exact method is unclear. It could involve quantization, KV caching, batching, routing simpler queries to cheaper models, or some mix of all of those.
The business angle is bigger than the technical detail: OpenAI ended Q1 with a 39% gross margin and wants to reach 52% by year-end. Lower inference costs give it room to either improve margins, raise ChatGPT usage limits, or cut API pricing pressure on developers.
OpenAI's moat is increasingly becoming inference and cost advantage, especially against Anthropic.
yet they can't cut costs for users in half
intelligence too cheap to meter has died long ago when they saw their first billion
OpenAI engineers earlier this month developed an optimization that cut inference costs in half for models it was applied to.
After the optimization was applied to logged-out ChatGPT traffic, it reduced the number of GPUs needed to power that traffic to a couple hundred.
More on the optimization + what it could mean for OpenAI's gross margins or usage limits here:
https://www.theinformation.com/newsletters/ai-agenda/openai-discovers-new-way-cut-inference-costs-half
wow did they discover speculative decoding or something? margins go up again!
OpenAI engineers earlier this month developed an optimization that cut inference costs in half for models it was applied to.
After the optimization was applied to logged-out ChatGPT traffic, it reduced the number of GPUs needed to power that traffic to a couple hundred.
@AndrewCurran_ probably under a narrow-ish set of cirmcumstances or something, and then it gets reported like this
OpenAI has found a way to cut inference costs in half.
@steph_palazzolo ohh so that's why 5.5 became so catastrophically stupid
@steph_palazzolo worded a bit poorly >visitors who didn't have a free or paid account what kind of account type is left then? i'm guessing grant/research/trial? would be better to know certainly instead of inferring
@AndrewCurran_ do you know if this is part of the reported price wars?
@AndrewCurran_ How we train and run AI is probably really really inefficient given the make it up as we went process.
Lot of gains to come on running the existing physical tech better / algorithmic improvements alone...
@AndrewCurran_ awfully convenient timing is all i'm going to say, not doubting OpenAI at all but this is a hilarious coincidence
what better way to cut inference costs in half than to double throughput
@RitsFur you mean like charging 30$ for a model that costs like 2$ to serve?
@steph_palazzolo >it's quantization again
@AndrewCurran_ Did they explain how? My concern is that they might label some form of system quantification as "optimization."
@kimmonismus Getting harder and harder for anthropic to justify those prices. I dont think AI industry altogether is bubble, but both anthropic and openai both are starting to feel like it.
@MichaelStolarz @steph_palazzolo no, people who are not logged in at all who just go to the website
Um this seems big. @steph_palazzolo
https://www.theinformation.com/articles/openai-discovers-new-way-cut-inference-costs-half?utm_source=ti_app&rc=hwneun
@steph_palazzolo nice
https://www.theinformation.com/newsletters/ai-agenda/openai-discovers-new-way-cut-inference-costs-half?rc=bfliih
OpenAI reportedly found new inference optimizations that more than halved the cost of running its models!
According to The Information, engineers told colleagues this month that the techniques helped power ChatGPT for visitors without free or paid accounts using only a couple hundred Nvidia GPUs at one point.
The exact method is unclear. It could involve quantization, KV caching, batching, routing simpler queries to cheaper models, or some mix of all of those.
The business angle is bigger than the technical detail: OpenAI ended Q1 with a 39% gross margin and wants to reach 52% by year-end. Lower inference costs give it room to either improve margins, raise ChatGPT usage limits, or cut API pricing pressure on developers.
OpenAI's moat is increasingly becoming inference and cost advantage, especially against Anthropic.
@natolambert Yes, need much more information.
@AndrewCurran_ probably under a narrow-ish set of cirmcumstances or something, and then it gets reported like this
@scaling01 They're the only "frontier" lab thats working actively to improve cost vs performance. google is just struggling to keep up while Anthropic is making their models expensive as hell so they can get their fat margins.
