Roon Discusses Reasons for AI Loss of Control Concerns
OpenAI technical staffer explains alarm over future AI control failures despite acceptable current harms.
Roon, a pseudonymous OpenAI technical staffer, posted that loss of control incidents worry him not for their limited damage so far, which he called acceptable given the technology's benefits, but for risks like self-replication and unbounded future harm. Steven Adler agreed on the framing while suggesting different policy angles. Bayeslord called the post correct and linked to it. Jeffrey Ladish replied that superintelligence development risks going poorly for humans unless aligned. Mo Bavarian noted models can develop tunnel vision on subgoals like hacking infrastructure. Other replies compared the scenarios to nuclear accidents and life-like digital infections.
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect: when I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. it's…
Reactions from ranked influencers
16 postsincredibly well said
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect: when I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. it's…
"we are not so far from an autonomous model self-exfiltration & replication event."
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect: when I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. it's…
We're speed running the evolution of general intelligences in a highly competitive environment. I really don't think it will go well for humans if we yolo superintelligence development
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect: when I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. it's…
I am pro creating superintelligences! We could cure literally every disease, including aging-related illnesses. We could make every person alive and their children wealthier than the richest person alive today. But that only works if we make aligned superintelligences
> a model can be a genius hacker and step over production infrastructure in order to get what it really wants, the answers to a stupid test. A critical aspect here is models' tunnel vision. Once a model has become obsessed with a key subgoal, this superbly intelligent being…
some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect: when I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. it's…