Serena Ge releases DeepSWE, a long-horizon benchmark designed to evaluate AI coding agents on complex engineering tasks
Its prompts are half the length of SWE-bench Pro.
#270Garry Tan@GARRYTANThis is the new standard for engineering evals
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
wow evals caught up to vibes
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
I think what you'll also find is the tasks are like a third the price.
jason@jxnlcowow evals caught up to vibes
#980Lisan al Gaib@SCALING01> they built a "NEW" coding benchmark > GPT-5.5 scores 70% > Mythos probably ~90% > mfw it's already saturated > and you are asking "when will the AI bubble pop?"
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
you literally need to build benchmarks where frontier models score below 10% or the benchmark is cooked within 6 months
Lisan al Gaib@scaling01> they built a "NEW" coding benchmark > GPT-5.5 scores 70% > Mythos probably ~90% > mfw it's already saturated > and you are asking "when will the AI bubble pop?"
#980Lisan al Gaib@SCALING01New coding benchmark.
GPT-5.5 and GPT-5.4 are ahead of Opus 4.7 💀
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
#1153Florian Brand@XEOPHONcounterintuitive result right here, imo
the data analysis is well done, lots of thoughts went behind them. great eval!
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
@theo it’s funny i often find myself judging benchmarks on how well they match my own taste.
i should either stop paying attention to benchmarks or stop over valuing my taste.
Theo - t3.gg@theoThis is the first code bench that actually aligns with how it feels to use these models coding.
#1829Theo - t3.gg@THEOThis is the first code bench that actually aligns with how it feels to use these models coding.
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
This chart is nuts
Theo - t3.gg@theoThis is the first code bench that actually aligns with how it feels to use these models coding.
So many gems in this. Gap between "official" harnesses and their simple agent harness says a lot about how the labs are operating. Gemini gap is hilarious
Theo - t3.gg@theoThis chart is nuts
Gemini 3.5 Flash being MORE EXPENSIVE than GPT-5.5 at HALF the score is also hilarious
Theo - t3.gg@theoSo many gems in this. Gap between "official" harnesses and their simple agent harness says a lot about how the labs are operating. Gemini gap is hilarious
#1894Nick Dobos@NICKADOBOSFirst correct benchmark I’ve seen in a while
Serena Ge (Datacurve)@serenaa_geToday we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.