A matched-token test challenges three web-agent methods' reported advantage

Researchers compared AWM, ASI and ReasoningBank with a vanilla agent free to spend the same token budget on more interaction steps. They say much of the methods' reported advantage disappeared on WebArena, and urge equal-token comparisons and multiple runs because single-run results can mislead. Another poster says they're also seeing scaffolding for its own sake often deliver poor performance per token.