Forethought's William MacAskill argues AI will outpace conceptual safety research, urging immediate near-term governance

Thanks Will. Useful response. I think the core crux is re: “You need to distinguish centralisation of decision-making and concentration of power. A liberal democracy has un-concentrated power and partly centralised decision-making.”

I take your point that centralized decision-making can in principle prevent concentration of power. But I think that this is very difficult to do well, akin to creating an organism that doesn’t get cancer. With each centralized decision there are opportunities and incentives for concentration of power, and ratchet effect in that direction.

Hence we’ve seen an extraordinary expansion of the regulatory and bureaucratic power of most western govts over the last century or two. And more specifically the key political lesson of the last half-century was that the *nominal* institutions of liberal democracy failed to prevent a lot of concentration of power in elite monocultures that diverged sharply from the public will (I like Lasch’s term “revolt of the elites” for this). International institutions take this even further in the direction of being unrepresentative: EU and UN officials are notoriously out of touch, and are thus a small group with a large amount of power, who justify their positions in part by fear-mongering about concentration of power by others (e.g. billionaires).

So how does one design an organism which doesn’t get cancer? There’s a kind of deep conceptual thinking which seems crucial here, and which you acknowledge you’re not doing. We mostly disagree about the scale of mistakes that occur in the absence of such thinking. The OpenAI board situation and FTX seem to me like important but ultimately small datapoints. The polarization of AI governance, and thing where Anthropic is now the hardest-racing lab, seem like bigger datapoints. We may well end up looking back at Anthropic as a cancerous (in the sense of power-seeking) growth that basically swallowed EA. (The actual credence you assign to that hypothesis is less important that the process of holding it as an emotionally live possibility.)

So when I point to the three mistakes above, I’m less saying that they are directly creating concentration of power, and more that they evince the kind of thinking which has a blind spot regarding gradual but cumulative takeovers (aka cancer). Yes, everyone might end up sharing an axiology for good reasons. But groups that identify themselves as “people with a shared axiology” (aka value aligned) are much more vulnerable to takeover than groups that coordinate around shared deontological principles and virtues even when they disagree about what outcomes are good. So puttin the diversity in the axiology is IMO structurally confused. Similarly, the entire field’s failure to grapple with the similarity between its current policy and the “maximize RSI” policy indicates that its proposed solutions are likely also confused. One example (which I don’t remember who specifically supported) is strong surveillance on all lab employees, which seems extremely prone to being captured by power-seeking actors.

Lastly: I reject the claim that there’s not enough time for deep conceptual progress. It’s odd that you state it so confidently! We’re all pretty confused about the future of AGI. Unfortunately AI safety people seem to be very vulnerable to the reasoning chain: “RSI soon is possible —> I should focus my efforts on that —> there’s no time for anything except {unprincipled intervention that will likely backfire}”. Panicking about the urgency has very literally created most of the urgency, and continues to do so.