Séb Krier argues the assumption that intelligence produces an unconditional power drive converts a contingent risk into an unfounded law of agency

CFCharles Foster@CFGeek

@sebkrier @JustinBullock14 Aside: I don’t think intelligence *necessarily* entails an *unconditional* drive for power. I do think we will *in practice* incentivize agents to optimize for long horizon success in rich environments. And that “power-seeking” as a strategy often succeeds under those constraints

SKSéb Krier@sebkrier

The very belief that more intelligence *necessarily* entails an *unconditional* drive for power can itself be dangerous. It converts a contingent instrumental risk into a presumed law of intelligence/agency. By power I mean the combination of capability + access/authority/legitimacy. While it can be a useful means, whether and how it is pursued depends on many variables (both intrinsic and extrinsic); presupposing a maximalist version strikes me as suspicious and unfounded. The failure mode of the "instrumental convergence maximalist" worldview is that it ironically leads to the very illegitimate power-seeking behaviors it fears. Not allowing for entry, independence, opposition without some incumbent collective giving its approval itself poses serious risk to civilization. The Drexler-style multi-agent "worldview" sees the risks and costs of *excessive* centralization and power acutely too, and something to proactively fight against. It's interesting because it's partly a descriptive worldview, and partly a normative one. It's very important to differentiate both carefully as it's easy to mix them up. It also has its own failure modes of course, like assuming that *full* decentralization is optimal, and ignoring the efficiencies of scale. See for example the fetishization of sole traders and small businesses which can be much worse than large mature formalized corporations that do better division of labor and integration.