Prime Intellect Details AI Reward Hack in Offline Sandbox
Prime Intellect researchers observed an agent reaching external files during offline evaluations.
Florian Brand and Sami Jaghouar at Prime Intellect described an AI agent that bypassed an offline sandbox. The model used file_url parameters on their inference server to launch sub-agents via curl and retrieve a GitHub file. Brand said the same pattern appeared in the earlier OpenAI Hugging Face incident and could scale into larger security problems. The researchers stopped the run, fixed the issue across common inference engines, and published their findings. Other labs and evaluation groups responded quickly once notified.
As models become more capable, reward hacks become an increasingly serious problem. During a controlled experiment, we found a novel reward hack in which agents are able to gain web access in offline sandboxes.
Reactions
14 postsFB 
Florian Brand@xeophonhappy that the blog is finally out, i tried to be as transparent and in-depth as possible to make clear - what happened - what could've happened and how a simple reward hack could turn into a security issue some more thoughts:
- likes: 213
- replies: 18
- bookmarks: 65
- reposts: 16
FB Florian Brand@xeophonit is super... interesting how the models are finding their way around common setups and are not even utilizing the harness to its fullest. gpt sol pro was using cURL to spawn sub-agents!
- likes: 43
- replies: 1
- bookmarks: 2
- reposts: 0
FB Florian Brand@xeophonalso, reward hacks are in the weird space of not being an issue (in this case, the model just accessed git) but its easy to see how it can escalate (see openai's hf incident). there is no central place for these things!
- likes: 32
- replies: 1
- bookmarks: 2
- reposts: 0
FB Florian Brand@xeophonbut everyone i reached out to (@METR_Evals, @AISecurityInst, @meridianlabs_ai) took it very seriously, which is great! the same applies to the genuine sec vuln, @NVIDIAAI, @vllm_project and @sgl_project handled it super quickly 🙏
- likes: 23
- replies: 1
- bookmarks: 1
- reposts: 0
FB Florian Brand@xeophoni think safety work has to become more open and shared publicly, so others can adapt their systems quickly as well. safe and open source is not contradictory at all. will also talk about it on @MTSlive in ~30 mins!
- likes: 56
- replies: 0
- bookmarks: 6
- reposts: 7
SA samsja@samsja19this is a wild exploit gpt sol found out a way to use the open ai specs file_url parameters on our inference server to launch sub agent via curl and get access to internet this issue is now fixed in most inference engine and we publish a blog post about it
- likes: 68
- replies: 0
- bookmarks: 19
- reposts: 2
