Serena Ge releases DeepSWE, a long-horizon benchmark designed to evaluate AI coding agents on complex engineering tasks
Its prompts are half the length of SWE-bench Pro.
#270Garry Tan@GARRYTANThis is the new standard for engineering evals
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
Honestly, looks about right, harness (mini-swe-agent) affinities aside Kimi is the closest to a mature autonomous SWE agent out of open models DS is weak and needs handholding (though has isolated strengths like debugging) a mark of good eval: stronger separation of top tier
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
wow evals caught up to vibes
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
I think what you'll also find is the tasks are like a third the price.
jason@jxnlcowow evals caught up to vibes
#980Lisan al Gaib@SCALING01> they built a "NEW" coding benchmark > GPT-5.5 scores 70% > Mythos probably ~90% > mfw it's already saturated > and you are asking "when will the AI bubble pop?"
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
you literally need to build benchmarks where frontier models score below 10% or the benchmark is cooked within 6 months
Lisan al Gaib@scaling01> they built a "NEW" coding benchmark > GPT-5.5 scores 70% > Mythos probably ~90% > mfw it's already saturated > and you are asking "when will the AI bubble pop?"
#980Lisan al Gaib@SCALING01New coding benchmark.
GPT-5.5 and GPT-5.4 are ahead of Opus 4.7 💀
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
#1153Florian Brand@XEOPHONcounterintuitive result right here, imo
the data analysis is well done, lots of thoughts went behind them. great eval!
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
#1496Chubby♨️@KIMMONISMUSIt's truly amazing to see how the general sentiment has shifted in favor of Codex.
I'm reading so many posts saying that Codex is really good now with GPT-5.5, and that Claude Code is regularly preferred.
(I've become a huge Codex fan myself).
At the same time, the new DeepSWE benchmark shows that GPT-5.5 is now ranked number one in this measurement as well.
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
#1711🍓🍓🍓@IRULETHEWORLDMOfinally, a great benchmark. now we just need one thousand more.
seriously though, this is great.
and we do need one thousand more.
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
@theo it’s funny i often find myself judging benchmarks on how well they match my own taste.
i should either stop paying attention to benchmarks or stop over valuing my taste.
Theo - t3.gg@theoThis is the first code bench that actually aligns with how it feels to use these models coding.
#1829Theo - t3.gg@THEOThis is the first code bench that actually aligns with how it feels to use these models coding.
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
This chart is nuts
Theo - t3.gg@theoThis is the first code bench that actually aligns with how it feels to use these models coding.
So many gems in this. Gap between "official" harnesses and their simple agent harness says a lot about how the labs are operating. Gemini gap is hilarious
Theo - t3.gg@theoThis chart is nuts
Gemini 3.5 Flash being MORE EXPENSIVE than GPT-5.5 at HALF the score is also hilarious
Theo - t3.gg@theoSo many gems in this. Gap between "official" harnesses and their simple agent harness says a lot about how the labs are operating. Gemini gap is hilarious
#1894Nick Dobos@NICKADOBOSFirst correct benchmark I’ve seen in a while
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
