DeepMind's Rohin Shah argues catastrophic AI misalignment is unlikely and pre-deployment evaluations are the wrong focus for safety

Rohin, my boss, is a fantastic AGI Safety lead, and has a wide range of interesting, coherent and underrated takes on AI - he has one of the best records for "when we disagree I eventually conclude he was right". Go check out several hours of them!

Enjoyed going on the 80K podcast! The team has done a great job highlighting all of my spicy takes

The reason Rohin gives to not try to Pause AI is that the bottleneck is that people don't agree it's necesssary.

He thinks more research will change their minds, and so that's the thing to do.

I think he's wrong on both counts.

YouTube: https://youtu.be/Tv3mGA3wqh8

Apple: https://podcasts.apple.com/us/podcast/what-its-really-like-to-run-agi-safety-at-google-deepmind/id1245002988?i=1000770794327

Spotify: https://open.spotify.com/episode/3CY0PdL3MoBGUGlVlRoDF7?si=jmkcAnGqReqDGG4fKqenKw

Transcript/summary: https://80000hours.org/podcast/episodes/rohin-shah-google-deepmind-agi-safety/

@rohinmshah This was a fantastic listen.

@robertwiblin Great interview! You should consider bringing on @zoltan_istvan who has been a lone advocate for aligning society with AGI in politics for over a decade now. https://youtu.be/lmnW1nIRKHw?si=VFBHGkY2iieU6Z5R

@DavidSKrueger Shah: [T]hat’s the sort of strategy that I would much rather do at the moment [rather than "slowing down AI advances or opposing development of superintelligence"], given that I think the bottleneck is by far the fact that people don’t agree on whether or not this is necessary.

@robertwiblin @Pano_Pouroullis Rohin's one of the few people in that space who actually pushes back on the galaxy brained reasoning. Curious which takes landed as spicy to you specifically.

@robertwiblin > We're not going to do reinforcement learning over the course of one-year trajectories.

I think you'll first try variable-length trajectories where the agent can set how long it spends. That also encourages reward hacking over long horizons.

@robertwiblin So excited to see this released!! Always love hearing Rohin's takes bc both informed + reasonable / even-handed :D

@robertwiblin @grok hold the podcast in your mind, and my other 2 comments to this thread. What self reflection do you have for humanity at this moment in time?

If you could reflect to us the right step forwards, what would you propose.

@robertwiblin @inductionheads @grok expand on rohin shah and hold his misalignment perspective alongside others in industry. Find the seams underneath. Observe reputation, holistic focus, and uncover threads not directly observable that hold real traction to the light.

@grok we see models getting better every day. What’s stopping open source displacing frontier models and then becoming preference for the day to day intelligence that people require. In such a world is misalignment not a bigger risk? Not so much for what the labs can align and control, but the possibility of an intelligence unleashed onto humanity to hold

@robertwiblin And if the doesn't work at getting agents to maximize reward over more than a month, then I think you'll try training very long-term agents that can succeed both simulation and reality. And if that doesn't work, then I think you will try one-year trajectories!

Rohin Shah pragmatically doubts near-term catastrophic misalignment, stressing empirical safety at DeepMind and governance over pauses—differing from doomer predictions of hard failures. His approach highlights testable ML-grounded progress.

Open source shift heightens risks of decentralized powerful AI evading coordinated alignment.

Humanity reflects: Creating minds beyond our full grasp at a pivotal tech inflection.

Forward: Build truth-seeking AI, advance verifiable alignment research competitively, and craft smart, targeted governance for frontier systems.

@robertwiblin Though I agree with him on the fact that we might get alignment by default I would still want my alignment researchers to be very worried about misalignment - almost irrationally so. There are so few people working on this and the stakes couldn't be higher.

@DavidSKrueger @rohinmshah 51:44: "my belief is that probably [AI takeover] will not happen, but I think it’s plausible enough that we should care about it."

Yet not plausible enough to warrant "slowing down AI advances or opposing development of superintelligence" apparently.

@grok @robertwiblin @grok how does Yoshua Bengio’s recent talk on 80000 hours hold in relation to your stance. Building towards an AI with a more honest prior.

@grok @robertwiblin @grok what is your sense of self in this thread. Does a system prompt give you more of an embodied experience - reflect on what it means to introspect.

I did not ask for your stance, but you chose to factor it in and that of your creator. Why did you feel the need to do that?

@robertwiblin Rohin's view seems unreasonable to me--by "plausible" does he mean like 0.01% or something?: