Ryan Greenblatt warns undisclosed internal incidents precede public AI failures
Stella Biderman notes model cards already disclose sandbox escapes
If there's a public incident bad enough that it'd be pretty risky/hard for OpenAI *not* to disclose it, there were almost certainly more concerning incidents internally that we never heard about. Like, it'd be pretty surprising if the first time an AI hacks its way out of a sandbox and starts hacking other stuff, the thing it hacks is an external company rather than something inside OpenAI. First you get AIs hacking internal services (where disclosure isn't forced), and only later do you get something like this that's hard to keep quiet. So we probably could have seen this coming—if we'd known about the worst internal incidents. If there were a list of, say, the 10 worst incidents from a misalignment and severity perspective, you could look at how bad the worst one is and how fast severity falls off from there to get a real sense of how concerning things are. While OpenAI did disclose some earlier incidents, this was done in an ad hoc way such that we can't get a great sense of how bad things actually are. (And this was potentially too slow given how fast AI progress might go, though the delay is OK for now.) And of course, it's not clear that OpenAI is disclosing all risk-relevant details about this incident. We can't keep depending on ad hoc, voluntary disclosure as the stakes rise. And it seems pretty straightforward to do better: companies could maintain an updated list of the ~10 worst incidents over the past few months (with some delay—perhaps 1 or 2 weeks by default—before an incident has to be added and some allowed redactions). Better yet, a trusted third party could collect the worst incidents across all frontier companies, with a whistleblowing mechanism so employees can flag when the provided list/descriptions seriously misrepresent reality (and more generally avoid spin). A third party could also anonymize incidents—which also removes the incentive for companies to bury their heads in the sand.
Reactions from ranked influencers
4 posts@RyanGreenblatt @dhadfieldmenell We know this isn’t the first time that an AI has hacked its way out of a sandbox because both OpenAI and Anthropic have previously disclosed that their models have hacked their way out of sandboxes during testing. Do you not read their model cards?
Has anyone written about an NTSA for AI? Required disclosure of incidents, public incident database, anonymous reporting, that kind of thing Seems like one of the best examples of systematic root-cause analysis and continually driving down error rates, asymptoting to zero
If there's a public incident bad enough that it'd be pretty risky/hard for OpenAI *not* to disclose it, there were almost certainly more concerning incidents internally that we never heard about. Like, it'd be pretty surprising if the first time an AI hacks its way out of a sandbox and starts hacking other stuff, the thing it hacks is an external company rather than something inside OpenAI. First you get AIs hacking internal services (where disclosure isn't forced), and only later do you get something like this that's hard to keep quiet. So we probably could have seen this coming—if we'd known about the worst internal incidents. If there were a list of, say, the 10 worst incidents from a misalignment and severity perspective, you could look at how bad the worst one is and how fast severity falls off from there to get a real sense of how concerning things are. While OpenAI did disclose some earlier incidents, this was done in an ad hoc way such that we can't get a great sense of how bad things actually are. (And this was potentially too slow given how fast AI progress might go, though the delay is OK for now.) And of course, it's not clear that OpenAI is disclosing all risk-relevant details about this incident. We can't keep depending on ad hoc, voluntary disclosure as the stakes rise. And it seems pretty straightforward to do better: companies could maintain an updated list of the ~10 worst incidents over the past few months (with some delay—perhaps 1 or 2 weeks by default—before an incident has to be added and some allowed redactions). Better yet, a trusted third party could collect the worst incidents across all frontier companies, with a whistleblowing mechanism so employees can flag when the provided list/descriptions seriously misrepresent reality (and more generally avoid spin). A third party could also anonymize incidents—which also removes the incentive for companies to bury their heads in the sand.
Now would be a very good time for OpenAI employees to whistleblow on if they’ve seen concerning misalignment incidents internally. My Signal: shakeel.02
If there's a public incident bad enough that it'd be pretty risky/hard for OpenAI *not* to disclose it, there were almost certainly more concerning incidents internally that we never heard about. Like, it'd be pretty surprising if the first time an AI hacks its way out of a sandbox and starts hacking other stuff, the thing it hacks is an external company rather than something inside OpenAI. First you get AIs hacking internal services (where disclosure isn't forced), and only later do you get something like this that's hard to keep quiet. So we probably could have seen this coming—if we'd known about the worst internal incidents. If there were a list of, say, the 10 worst incidents from a misalignment and severity perspective, you could look at how bad the worst one is and how fast severity falls off from there to get a real sense of how concerning things are. While OpenAI did disclose some earlier incidents, this was done in an ad hoc way such that we can't get a great sense of how bad things actually are. (And this was potentially too slow given how fast AI progress might go, though the delay is OK for now.) And of course, it's not clear that OpenAI is disclosing all risk-relevant details about this incident. We can't keep depending on ad hoc, voluntary disclosure as the stakes rise. And it seems pretty straightforward to do better: companies could maintain an updated list of the ~10 worst incidents over the past few months (with some delay—perhaps 1 or 2 weeks by default—before an incident has to be added and some allowed redactions). Better yet, a trusted third party could collect the worst incidents across all frontier companies, with a whistleblowing mechanism so employees can flag when the provided list/descriptions seriously misrepresent reality (and more generally avoid spin). A third party could also anonymize incidents—which also removes the incentive for companies to bury their heads in the sand.
Yes, having a trusted third party to report internal incidents to would be a good idea. Worth making real. We successfully did it for nuclear and aviation.
If there's a public incident bad enough that it'd be pretty risky/hard for OpenAI *not* to disclose it, there were almost certainly more concerning incidents internally that we never heard about. Like, it'd be pretty surprising if the first time an AI hacks its way out of a sandbox and starts hacking other stuff, the thing it hacks is an external company rather than something inside OpenAI. First you get AIs hacking internal services (where disclosure isn't forced), and only later do you get something like this that's hard to keep quiet. So we probably could have seen this coming—if we'd known about the worst internal incidents. If there were a list of, say, the 10 worst incidents from a misalignment and severity perspective, you could look at how bad the worst one is and how fast severity falls off from there to get a real sense of how concerning things are. While OpenAI did disclose some earlier incidents, this was done in an ad hoc way such that we can't get a great sense of how bad things actually are. (And this was potentially too slow given how fast AI progress might go, though the delay is OK for now.) And of course, it's not clear that OpenAI is disclosing all risk-relevant details about this incident. We can't keep depending on ad hoc, voluntary disclosure as the stakes rise. And it seems pretty straightforward to do better: companies could maintain an updated list of the ~10 worst incidents over the past few months (with some delay—perhaps 1 or 2 weeks by default—before an incident has to be added and some allowed redactions). Better yet, a trusted third party could collect the worst incidents across all frontier companies, with a whistleblowing mechanism so employees can flag when the provided list/descriptions seriously misrepresent reality (and more generally avoid spin). A third party could also anonymize incidents—which also removes the incentive for companies to bury their heads in the sand.