• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
AI
Report

The case for coding-agent benchmarks that match developer task flows

A post describes a JetBrains Research paper contrasting single-issue tests with sessions spanning questions, planning, reviews and edits.

1 Source, 22h ago, first seen 22h ago

TLDR

A post describes a JetBrains Research paper arguing that benchmarks should reflect how coding-agent users switch tasks mid-session. It says researchers studied 4,782 real agent sessions, where feature and bug-fix requests were well under half, compared with 93.8% of SWE-Bench Pro tasks. The paper’s SWE-TaskFlow splits issues into turns and adds checkable code questions; on 700 SWE-Bench Pro tasks, the post says that split roughly doubled agent cost without a stable change in success rate.

Combined views

—

1 Source, first seen 22h ago

— likes— comments— saves— reposts

Combined views

—

1 Source, first seen 22h ago

— likes— comments— saves— reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Featured Source

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

1 Source

Rohan Paul@rohanpaul_aiNew JetBrains Research paper argues that coding-agent benchmarks should copy how their real users actually work, because real sessions mix and switch task types across many messages. Real developers using coding agents ask questions, plan, review, and switch tasks mid-session, but benchmarks like SWE-Bench Pro hand the agent 1 detailed issue and grade 1 final patch. In that benchmark, new features and bug fixes make up 93.8% of tasks. JetBrains researchers studied 4,782 real agent sessions, where features and bug fixes were well under half of requests. In longer sessions, users kept switching between questions, planning, reviews, and edits. They built SWE-TaskFlow, which splits each benchmark issue into several turns and adds checkable questions about the code. On 700 SWE-Bench Pro tasks, splitting roughly doubled agent cost with no stable change in success rate. If you evaluate coding agents, first measure what your own users ask for, then shape the benchmark's conversation to match it. – arxiv. org/abs/2610.09633 Title: "Coding-Agent Benchmarks Should Match Their Users' Task Flows"22h
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    1 Source

    Rohan Paul@rohanpaul_aiNew JetBrains Research paper argues that coding-agent benchmarks should copy how their real users actually work, because real sessions mix and switch task types across many messages. Real developers using coding agents ask questions, plan, review, and switch tasks mid-session, but benchmarks like SWE-Bench Pro hand the agent 1 detailed issue and grade 1 final patch. In that benchmark, new features and bug fixes make up 93.8% of tasks. JetBrains researchers studied 4,782 real agent sessions, where features and bug fixes were well under half of requests. In longer sessions, users kept switching between questions, planning, reviews, and edits. They built SWE-TaskFlow, which splits each benchmark issue into several turns and adds checkable questions about the code. On 700 SWE-Bench Pro tasks, splitting roughly doubled agent cost with no stable change in success rate. If you evaluate coding agents, first measure what your own users ask for, then shape the benchmark's conversation to match it. – arxiv. org/abs/2610.09633 Title: "Coding-Agent Benchmarks Should Match Their Users' Task Flows"22h
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet