BreakingWeb tests browser agents with altered web-app tasks

BreakingWeb pairs 519 tasks across seven self-hosted web apps, comparing clean runs with runs that include controlled, recoverable changes. Its authors report that text-mode agents’ pass rates fell by 17.9 to 27.6 percentage points under intervention. On a separate human panel, the pass rate for warm human participants fell by 5.7 points. They say 75% of classified text-mode intervention failures ended with agents declaring success without reaching the goal.