Reactions from ranked influencers
3 posts
Kimi-K3 is the strongest open-weight model on LisanBench and it outperforms Gemini 3.1 Pro (high), but still loses to Opus 4.7 xhigh at similar token usage. Overall it ranks 5th in the standard metric, and 4th in our difficulty weighted metric, the best-of-3 metric and in terms of ratio of valid transitions. It outperforms GPT-5.5 and Opus 4.8, but uses 2-3x more tokens. I think this is consistent with what we see on other benchmarks. Kimi-K3 is a genuinely strong model that needs 1 or 2 more iterations to improve reasoning efficiency to truly catch up to the frontier. A few more things I want to note: - the search behavior seems to be distinctively different from all other Frontier Models (image 2) - it uses some highly obscure or outdated words, which are not in my standard dictionary. this is the problem with fixing the dictionary, you will always miss words. If we allowed those words, the score would increase by ~13%, which wouldn't change the ranking but would slightly improve reasoning efficiency. With the same "corrected" scoring Fable 5 and Opus 4.8 would increase by 1.21%, GPT-5.5 by 6.86%, GPT-5.6-Sol by 7.93%. So Kimi-K3 is hit a bit heavier by the dictionary. - these two facts made it a little bit sus, as we have seen some forms of "soft cheating" before with Opus 4.6 and Sonnet 4.6 with the bridge pattern, which I have written an article about. However, Kimi finds a new way to increase scores. I wouldn't call it cheating, since the rules allow it. The TLDR of Kimi's strategy is, that it finds clusters of words that are all edit-distance 1 and sweeps through them by changing only 1 letter at the same position over and over like this: - may → bay → cay → day → fay → gay → hay → jay → lay → nay → pay → ray → way → yay → say GPT-5.6-Sol calls these clusters wildcard families (derived from database wildcard queries). Kimi-K3 does this to an excessive extent (image 3), such that ~35% of all its valid transitions occur in such wildcard families. Every other model is below 2% on that metric. Other models change the word stem much more frequently, either by deleting or adding a letter or substituting at a different position.
Kimi-K3, like K2, has a delightfully large active vocabulary
Kimi-K3 is the strongest open-weight model on LisanBench and it outperforms Gemini 3.1 Pro (high), but still loses to Opus 4.7 xhigh at similar token usage. Overall it ranks 5th in the standard metric, and 4th in our difficulty weighted metric, the best-of-3 metric and in terms of ratio of valid transitions. It outperforms GPT-5.5 and Opus 4.8, but uses 2-3x more tokens. I think this is consistent with what we see on other benchmarks. Kimi-K3 is a genuinely strong model that needs 1 or 2 more iterations to improve reasoning efficiency to truly catch up to the frontier. A few more things I want to note: - the search behavior seems to be distinctively different from all other Frontier Models (image 2) - it uses some highly obscure or outdated words, which are not in my standard dictionary. this is the problem with fixing the dictionary, you will always miss words. If we allowed those words, the score would increase by ~13%, which wouldn't change the ranking but would slightly improve reasoning efficiency. With the same "corrected" scoring Fable 5 and Opus 4.8 would increase by 1.21%, GPT-5.5 by 6.86%, GPT-5.6-Sol by 7.93%. So Kimi-K3 is hit a bit heavier by the dictionary. - these two facts made it a little bit sus, as we have seen some forms of "soft cheating" before with Opus 4.6 and Sonnet 4.6 with the bridge pattern, which I have written an article about. However, Kimi finds a new way to increase scores. I wouldn't call it cheating, since the rules allow it. The TLDR of Kimi's strategy is, that it finds clusters of words that are all edit-distance 1 and sweeps through them by changing only 1 letter at the same position over and over like this: - may → bay → cay → day → fay → gay → hay → jay → lay → nay → pay → ray → way → yay → say GPT-5.6-Sol calls these clusters wildcard families (derived from database wildcard queries). Kimi-K3 does this to an excessive extent (image 3), such that ~35% of all its valid transitions occur in such wildcard families. Every other model is below 2% on that metric. Other models change the word stem much more frequently, either by deleting or adding a letter or substituting at a different position.
benchmarking it was also more expensive than Opus 4.8 and GPT-5.6 Sol