(DeepSeek V3.2, GLM 4.7, and Kimi K2.5) prompted to resolve issues in real public code repositories. All trajectories include interleaved reasoning and tool calls.
2/4
Artificial Analysis just announced AgentPerf, the industry’s first agentic AI benchmark. The benchmark uses several real-world agentic use trajectories and employs OpenCode agentic harness using three top open-source models with reasoning enabled
1/4
Users criticized the AgentPerf benchmark launch for relying on a brutally bad or outdated model choice and questioned whether the post was serious.
Artificial Analysis just announced AgentPerf, the industry’s first agentic AI benchmark. The benchmark uses several real-world agentic use trajectories and employs OpenCode agentic harness using three top open-source models with reasoning enabled
1/4
Among other findings the benchmarks shows that NVIDIA's GB300 outperforms H200 by a whopping factor of 20 in terms of running concurrent agents per megawatt.
3/4
AA methodology: https://artificialanalysis.ai/methodology/agentperf Nvidia technical blog: https://developer.nvidia.com/blog/nvidia-achieves-leading-agentic-coding-performance-on-first-agentic-ai-benchmark/
4/4
@tunguz The model choice is brutally bad/outdated. Is this a serious post? 😅