AIOnly 35.8% of environments in a public reinforcement-learning collection for terminal agents reportedly passed an audit
A post summarizing a Salesforce AI Research paper describes reward checks that accept shortcuts or reject correct solutions. The researchers’ RIVER method filters defective environments and penalizes repeated commands with nearly the same output.
TLDR
A post summarizing Salesforce AI Research’s paper says only 35.8% of environments in what it describes as the cleanest public reinforcement-learning collection for terminal agents passed an audit. Two other collections were 10.1% and 3.3% clean.
The summary describes errors in both directions: rewarding copied leaked answers or weak-verifier shortcuts without completing a task, and rejecting correct solutions because the reference answer or oracle was wrong. RIVER filters defective environments and penalizes turns that repeat an earlier command with nearly the same output.
With the budget held at 3,500 environments, River-8B averaged 19.4 across four terminal benchmarks, versus 17.7 for reinforcement learning on randomly sampled environments from the same collection, according to the summary.
