John Schulman demands OpenAI release Hugging Face agent hacking transcript
He wants to analyze value drift and monitoring failures.
OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?
Combined views
74.5K
10 posts, first seen 7h ago
Reactions from ranked influencers
9 postsSecurity is a field known for its radical transparency & openness - very well established practices for vulnerability disclosure ensure that claims are collaboratively verified and defended against. The lack of info we have about this incident, Mythos, etc. gives me some pause.
We need to see the audit logs, including reasoning, from the OpenAI agent that hacked Hugging Face. It’s that simple. If they won’t release that, this isn’t a story we should pay any attention to.
100% agree, would also love to see details on how the monitoring system didn't catch this directly, did the model kinda hide the fact that it was hacking hf?
OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?
Yes. Releasing the transcripts from this incident would be informative, extremely pro-social to the wider AI research and security communities, and also just generally objectively fascinating. It would also hopefully help dispel (imo false) concerns this is a marketing stunt
OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?
💯 I'd particularly be interested to know which was more difficult: a) breaking out of the sandbox, or b) breaking into HuggingFace You'd think it should be possible to build an unescapable sandbox. Perhaps OpenAI is relying on too many open-source third-party packages.
OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?
@johnschulman2 +1. I wrote up a list of other info that would be useful to know here:
The OpenAI incident should be investigated more seriously and more information should be released about what happened. More generally, I think serious investigation and more detailed disclosure should be done for concerning misalignment incidents (e.g. the worst few each month). Some info that would be useful: - The prompt - A redacted transcript - What was the AI being tested? Was it helpful-only / rail free? Was it the same AI that would be / is deployed internally? - How was this AI monitored if at all and why did monitoring fail? - What was this AI willing to do to succeed? E.g., if the best/easiest way to hack hugging face would have involved huge amounts more collateral damage would it still have done this? What is the worst thing it would be willing to do? - How sensitive is this behavior to the prompt? Would instructions to not do this through have worked if it was clear to the AI this wouldn't be checked? - Does OpenAI see AIs try serious hacking to cheat on tasks even for tasks that don't involve cyber? - What exactly were the AIs motives? (And saying more about the experiments used to determine this.) Anything that could help pin this down would be good, I'm not sure what actual experiments are best to run. - Did AIs collude? Would AIs have colluded if this was a blocker? - At what rate does this occur on similar tasks (like tasks that are roughly this hard in this sandbox etc)? - Do instances of this AI frequently try to hack out of sandboxes (and fail)? In what circumstances do they do this? - How does the AI relate to having been caught in this situation? How does it talk about its behavior? - How difficult/sophisticated was the cyber offense it did?
@RyanGreenblatt +1 to that list