PROBE and the case for proactive coding-agent benchmarks

A commenter argues that many coding benchmarks in the past year have made SWE-bench-style tasks harder without changing their basic format. They praise PROBE's focus on work people still do, such as writing issues. Another post describes a study of proactive bug-finding agents, saying its recall and precision are measured against a holdout set of human-found bugs, while “findings” counts the total bugs found.