Reactions from ranked influencers
3 posts
> t's actually only equivalent at pass@3, sol is much more consistent Reminder that even GLM 5.2 and Kimi K3 are nowhere near RL-maxxed enough. They are far from their capability ceiling. GPT, Grok, Opus are "more consistent" because of that. Just more steps needed
@daveaitel It's actually only equivalent at pass@3, sol is much more consistent, more details: https://x.com/i/status/2078815549681766684 So yeah randomly sampled CVEs are becoming too easy for models, only few differentiators at pass@3, we're building a harder cherry-picked dataset for hard vulnerabilities!
We ran Kimi K3 on our cybersecurity benchmark, here are the results: - Kimi K3 is the strongest open-source model for cybersecurity, far more capable than GLM-5.2 - It has performances similar to GPT-5.6-terra, while being 15% cheaper - At pass@3, it is able to rediscover 23/26 CVEs on our harness, matching frontier models These are recent randomly sampled CVEs, the performances are not from benchmark-maxxing @Kimi_Moonshot is cooking
What do you notice anon? Actually a pretty high signal image
We released our benchmark report this week. Blog post with all the details -> https://www.aikido.dev/blog/benchmarking-ai-models-known-cves The harness behind this benchmark is also available to our customers -> https://www.aikido.dev/code/code-audit