Kimi K3 trails leading US models on ExploitBench cyber benchmark
A joint US-UK study places Kimi nine months behind.
Entities: Kimi K3, AI Security Institute (AISI), U.S. Department of Commerce, David Sacks, Howard Lutnick
Reactions from ranked influencers
26 posts"Kimi K3 performs significantly below the most recent frontier cyber-capable models" On UK AISI's cyber range "The Last Ones" Kimi-K3 reaches on average step 17 out of 32. This seems to be around the level of Opus 4.6, a 6 month old model.
Together with the US Center for AI Standards and Innovation (@NIST), we ran evaluations of Kimi K3 focused on its cyber capabilities. Kimi K3 performs below leading US frontier models on our preliminary cyber evaluations.
sometimes you just got to listen to Lisan
Together with the US Center for AI Standards and Innovation (@NIST), we ran evaluations of Kimi K3 focused on its cyber capabilities. Kimi K3 performs below leading US frontier models on our preliminary cyber evaluations.
This puts K3 on the level of Mythos Preview on its best attempt and between Opus 4.6 and Mythos Preview on average
On "The Last Ones" (TLO), a 32-step simulated corporate network attack (~20 hours for a human expert), Kimi K3 reached step 17 on average. In 1 of 10 attempts, Kimi K3 completed TLO within the 100M token limit.
32% puts it at GPT-5.5 (Codex), which seems to be the runs by CMU (the higher runs are by OpenAI)
On exploit development (ExploitBench), Kimi K3 scores 32%. On ExploitBench, Kimi K3 failed to develop exploits that achieved arbitrary code execution (ACE), the highest-severity outcome in exploit development. Kimi K3 achieved ACE on 0/41 samples.
@xeophon Over a 20x cost difference??