OpenAI internal optimization cuts inference costs by half, running logged-out ChatGPT traffic on a couple hundred GPUs

Observers speculate the undisclosed method relies on speculative decoding