Reactions from ranked influencers
11 posts
Have Chinese AI Models Caught Up to the US Frontier? I have spent the last 2 days writing this article. It should settle the debate once and for all. https://open.substack.com/pub/scaling01/p/have-chinese-ai-models-caught-up
One thing that I should clarify (which my moot stochasm pointed out) is that I first say that Artificial Intelligence Index shouldn't be used to estimate the true strength of models, but then I use it to forecast Kimi-K3's ECI. That's a bit silly. Both indexes are highly correlated, but the problem with the Artificial Analysis Index is that it doesn't separate the best models as well as ECI. The observation basically was that Chinese models are overrated on AAI and this is exactly what we see when we fit a single applied logit-linear regression. Only 36.7% of Chinese models are above it, but 57.5% of US models. With that single regression line the Kimi-K3 ECI forecast is 158.33 with 80% CI of [154.49, 162.02]. However, this is likely an overestimate. What we can do to improve this model is separate US and Chinese models and fit two separate trendlines. That new methodology gives us slightly better fits and predicts an ECI of 156.35 for Kimi-K3, which is also much closer to the preliminary observed value of 155.53. If we consider that Kimi-K3 is a much larger model and and an outlier, than we should also move our prediction slightly above this mean of 156.35.
Have Chinese AI Models Caught Up to the US Frontier? I have spent the last 2 days writing this article. It should settle the debate once and for all. https://open.substack.com/pub/scaling01/p/have-chinese-ai-models-caught-up
I also want to thank Artificial Analysis and EpochAI for having such great benchmarks and publicly available data
I estimated both the forward- and backward-looking gap between Chinese and US frontier models Kimi-K3 is currently 4.37 to 5.29 months behind US frontier models (backward-looking) Chinese models are projected to catch up Mythos-Preview by end of December 2026 (forward-looking), meaning they are behind by 8.57 to 8.83 months The gap between Chinese and US frontier models also seems to be widening These are all just public figures that likely understate the extent to which Chinese models are behind, as they don't consider safety testing, reasoning-efficiency, model sizes, available compute and more Furthermore, it's important to highlight that this is speaking in terms of overall capability across all domains. This estimate does not consider individual domains.
Have Chinese AI Models Caught Up to the US Frontier? I have spent the last 2 days writing this article. It should settle the debate once and for all. https://open.substack.com/pub/scaling01/p/have-chinese-ai-models-caught-up
i could've probably spent an additional day to refine and proof-read it but i think it's good maybe I will share some more thoughts or refine it when I upload it on X in a couple of days
@teortaxesTex The problem with *all* IRT applications is that it’s relatively noisy. It is elegant, but you have to look at the error bars and take the results with two big scoops of salt. Adding or removing just one of the scores can have 1-2 pts of impact, which is massive