Reaction
Audit of Zapier's AutomationBench verifiers reportedly found 206 bugs
A post says Parsave audited all 600 public tasks; regrading 1,235 Kimi K3 runs after fixes changed 27.9% of grades.
TLDR
A post says Parsave audited the verifiers for all 600 public tasks in Zapier's AutomationBench. Agents wrote realistic wrong answers to try to fool each verifier, and human review confirmed 206 bugs, according to the post. It says AutomationBench Verified fixed all 206; regrading 1,235 Kimi K3 runs with the fixed verifiers changed 27.9% of grades.
Combined views
6.5K
4 Sources, first seen ago