Commentary on X
It is due to math and more specifically a specific math benchmark which feels like an outlier, "integrals with game" see it. I feel this is likely a parsing issue if LLM as a judge is not used here, I am not sure though. If we remove math, it is within the margin of error with the rest of the top. I think this benchmark is saturated now and doesn't actually describe anything at this point. It was useful in the past.
This is incredible! Kimi K3 is not just competitive, it is trading wins with the best frontier models while apparently demolishing the model closest to it on price. If the independent benchmarks and token efficiency numbers hold up, this may be one of the biggest model launches of the year for sure!