HackBench debuts with every tested AI monitor missing some reward hacks

Finding the offending commands

HackBench gives each executable command in a transcript a numbered label. A monitor must return the indices of the commands involved in the hack, not merely flag the transcript as suspicious. Across the benchmark’s 169 trajectories, the team reports a median of more than 1 million characters and 264 commands per trajectory, but a median of just one hacked command.

The team says 41 trajectories contain naturally occurring hacks and 128 contain synthetic hacks. It reports that agents rarely hacked in preliminary experiments when explicitly told not to. To build the synthetic set, the researchers inserted hack commands into otherwise unmodified trajectories and updated affected later observations. They used GLM-5.2 for those changes, Claude Opus 4.8 and GPT-5.5 to audit the resulting trajectories, and manual annotation where the auditors disagreed.

For scoring, the team compares each monitor’s selected command indices with its audited labels. F1 combines precision, which measures how many flagged commands belong to that set, with recall, which measures how many labeled commands the monitor recovered. The team calculated those measures for each trajectory and averaged them across the benchmark. It reports that Muse Wonder followed Claude Opus 5 at 61.47 F1, while every other tested model scored below 53. Those scores describe performance on HackBench’s tested trajectories, most of which contain injected hacks.