Report
Ramp says SWE-Bench v2’s top scores bunch up, but compute use varies widely
Ramp says its new benchmark draws 500 tasks from production work by more than 200 engineers.
TLDR
In Ramp’s results, GPT-6.2 Astra solves 76.4% of tasks versus Gemini Carbon’s 75.4%, while averaging 6.5 minutes per task against Carbon’s 26.3. A near tie on score, not on the stopwatch. Ramp says using models’ native agent software generally boosts solve rates and costs versus Bash-only tools, though the effect varies by model.
Combined views
—
5 Sources, first seen ago

Comments