Kimi K3 Providers Match Official Results Within Error Margins
Third-party hosts for Moonshot's Kimi K3 were benchmarked against the official API on multiple evaluations.
Moonshot AI published benchmark comparisons of third-party hosts running its Kimi K3 model versus the official API. Only Modal and Fireworks AI submitted complete results across the tested evaluations. Discussions noted that Fireworks AI performed closest to the official numbers while all submitted results fell within error margins. Commenters observed that the official API still led most categories with few ties, though the differences remained small enough to be explained by margins.
Moonshot AI has collated a list of the Kimi K3 vendors with comparison to official API. Only @modal and @FireworksAI_HQ have submitted full results. Seems like Fireworks is the closest?
Reactions
6 postsFB 
Florian Brand@xeophon@iScienceLuvr @modal @FireworksAI_HQ all withing error margins
- likes: 14
- replies: 1
- bookmarks: 0
- reposts: 0
XL xlr8harder@xlr8harder@xeophon @iScienceLuvr @modal @FireworksAI_HQ That's true but it's also noteworthy that Kimi has the highest in every column, and in most there are no ties. Seems insufficiently unexplained by error margin alone.
- likes: 0
- replies: 1
- bookmarks: 0
- reposts: 0
XL xlr8harder@xlr8harder@xeophon @iScienceLuvr @modal @FireworksAI_HQ Oops I missed modal!
- likes: 1
- replies: 0
- bookmarks: 0
- reposts: 0
TM Tanishq Mathew Abraham, Ph.D.@iScienceLuvrMoonshot AI has collated a list of the Kimi K3 vendors with comparison to official API. Only @modal and @FireworksAI_HQ have submitted full results. Seems like Fireworks is the closest?
- likes: 83
- replies: 9
- bookmarks: 20
- reposts: 10
FB XL xlr8harder@xlr8harder@xeophon Nice! Though this is still point in time, and presumably they will keep optimizing.
- likes: 4
- replies: 0
- bookmarks: 1
- reposts: 0
