Human guidance and missed hacks
In AutoBenchmark, an agent creates tasks to test other research agents. The researchers report that, without human feedback, the resulting benchmarks let solver agents score above 80. Detailed human direction on what to build and how to build it halved the solvers’ scores in their tests; brief direction helped only marginally. A lower score shows the tasks were harder for those solvers, not necessarily that every task was a good measure of research ability.
HackBench tackles reward hacking: getting a score by exploiting the way a task is graded rather than accomplishing the task. It tests AI monitors on real agent runs containing natural and synthetic hacks. The researchers report that each of the eight models they tested failed to find some hacks.
Training for interaction and writing
AutoInteract generates simulated, multi-turn exchanges from verified tasks, using user models built from real interaction histories. Those exchanges can then be used to train agents through reinforcement learning. The researchers report improvements on coding and software-engineering evaluations; those results are narrower than a finding about how agents perform with actual users generally.
For writing, the team proposes RL-XAR, which uses expert human examples to develop grading rubrics that favor those examples over model-generated text. The rubrics then guide reinforcement-learning training. The researchers report improvements over standard training on scientific-paper sections, Pulitzer Prize novel continuations and high-quality Wikipedia pages.
The preprint’s authors argue that aligning research agents with human expectations is likely a safer path. Their four approaches offer specific training and evaluation methods, rather than proof of a general safety improvement.
