/Tech29d ago

DeepMind's Séb Krier questions instrumental convergence and asks for literature explaining the logic of AI power-seeking

David Manheim asked Krier to specify his exact objections.

7601333

#667

Original post

Séb Krier@sebkrier#667inTech

@davidmanheim @eschatropic @JeffLadish What is the best single piece/article you have explaining the logic here, apart from the Christiano LW post or Carlsmith 2022 paper? I continue to be skeptical of the rationalist account of power seeking and instrumental convergence.

David Manheim@davidmanheim

@eschatropic @JeffLadish @sebkrier 1. Because optimization pressure means the places it can fail are ones that will fail. 2. Yes, it's a convergent instrumental goal. Bengio thinks he could avoid it; I'm skeptical, but he agrees that it won't happen by default. 3. Because the systems will get stronger over time.

11:57 AM · May 31, 2026 · 80 Views

Sentiment

Sentiment building, check back later.

Cluster Engagement

Digg Deeper

No Digg Deeper questions have been answered for this story yet.

IFANYONEBUILDSIT.COMVia

Posts from X

Most Activity

VIEWS52REPLIES1

Séb Krier@sebkrier

@davidmanheim @eschatropic @JeffLadish A bunch of different things but don't think I'll have the time to elaborate on all aspects here, I'm mostly interested in collecting/re-exploring the key pieces about the worldview.

David Manheim@davidmanheim

@sebkrier @eschatropic @JeffLadish I'm not sure, but I don't really understand what's being questioned about the thesis; are you arguing that systems might not have goals, or that they might not want to achieve them, or that generally valuable resources and power might not help with doing so?

29d5210

LIKES3

Jeffrey Ladish@JeffLadish

@sebkrier @davidmanheim @eschatropic I have some interest in writing this up

Séb Krier@sebkrier

29d4230

Séb Krier@sebkrier

@davidmanheim @eschatropic @JeffLadish Yes I don't think it's particularly good, or makes any arguments really - just claims stuff

David Manheim@davidmanheim

@sebkrier @eschatropic @JeffLadish Yeah, I don't have anything in my pocket on this, and I'm guessing you're not going to think the relevant Appendix to IABIED is a sufficient argument, given that it's just making the basic point clearly; https://ifanyonebuildsit.com/5/instrumental-convergence

29d4710

David Manheim@davidmanheim

29d14