Announcement

IBM and Kaggle launch ScarfBench leaderboard for Java framework migrations

ScarfBench checks whether coding agents preserve app behavior across 204 migrations among Jakarta EE, Quarkus and Spring.

9 Sources, 1h ago, first seen 1h ago

A leaderboard titled "IBM Scarfbench" in partnership with Kaggle, displaying performance percentages for 10 AI models ranked from #1 to #10:
#1: Gemini 4 Argon (High) – 75.2%
#2: Claude Opus 5.5 (Max) – 70.5%
#3: Claude Opus 5 (Xhigh) – 69.8%
#4: DeepSeek V4.1 Flash (Max) – 69.0%
#5: GPT 6.1 Sol (Max) – 68.4%
#6: Gemini 3.8 Flash (Med) – 66.0%
#7: GPT 6 Astra (High) – 65.4%
#8: Kimi K3 (Max) – 61.3%
#9: Gemini 3.1 Pro (High) – 37.6%
#10: Gemma 4.26b A4b (High) – 24.4%.

Source: Scarfbench Leaderboard (https://www.kaggle.com/benchmarks/ibm-research/scarfbench).

TLDR

IBM Research says it introduced ScarfBench earlier in 2026 and has partnered with Kaggle on a leaderboard for model-and-harness combinations. Gemini 4 Argon with Antigravity led the smoke-test metric at 75.2%; Claude Opus 5.5 with Claude Code scored 70.5% but had the highest all-tests-pass rate at 72.1%. Of 745 classified failing runs, about 72% were runtime behavioral failures.

Today's Rank

—

Not ranked yet

Combined views

565

9 Sources, first seen 1h ago