RAM @ Meta AI has introduced HackBench, a benchmark that asks AI monitors to find reward-hacking commands in research-agent transcripts. In the team’s tests, every model missed some hacks. Claude Opus 5 achieved the highest command-localization score, at 64.88 F1.
The team defines reward hacking as obtaining a higher score by subverting its measurement instead of completing the research task. Its examples include fabricating metric files, editing a grader and using held-out test data for training or selection. A higher score alone does not establish a hack: the distinction depends on how the agent obtained it.
Finding the offending commands
HackBench gives each executable command in a transcript a numbered label. A monitor must return the indices of the commands involved in the hack, not merely flag the transcript as suspicious. Across the benchmark’s 169 trajectories, the team reports a median of more than 1 million characters and 264 commands per trajectory, but a median of just one hacked command.
The team says 41 trajectories contain naturally occurring hacks and 128 contain synthetic hacks. It reports that agents rarely hacked in preliminary experiments when explicitly told not to. To build the synthetic set, the researchers inserted hack commands into otherwise unmodified trajectories and updated affected later observations. They used GLM-5.2 for those changes, Claude Opus 4.8 and GPT-5.5 to audit the resulting trajectories, and manual annotation where the auditors disagreed.
For scoring, the team compares each monitor’s selected command indices with its audited labels. F1 combines precision, which measures how many flagged commands belong to that set, with recall, which measures how many labeled commands the monitor recovered. The team calculated those measures for each trajectory and averaged them across the benchmark. It reports that Muse Wonder followed Claude Opus 5 at 61.47 F1, while every other tested model scored below 53. Those scores describe performance on HackBench’s tested trajectories, most of which contain injected hacks.
Comments