My reasoning behind wanting to stop superalignment research: - Principal-agent alignment is neither necessary nor sufficient for safety and ecosystemic health. I think the vast majority of our problems will come from scenarios involving systemic harms, negligence, or malice instead of situations in which someone benevolent was exercising what would be considered “best technical practices” at the time but non-foreseeably loses control of their AI. - I think that current alignment, control, and containment strategies are actually pretty good and there is a big incentive for ML people to underemphasize the effectiveness of these tools to justify their existence. If you’re willing to pay a safety tax and are not “move fast and break things,” existing best practices can make systems pretty robustly safe. - Superalignment is pretty safety washed and is touted by big companies to justify their ambitions to build the superintelligence. - Solving superalignment would be a huge boon and would consolidate enormous power in big tech. This is itself a risk factor. - Jevons paradox — it’s easy to see how lowering the perceived risk of building superintelligence would make more companies choose to try.

For clarity:

Do you believe that automated alignment, scalable oversight and superalignment are bad because:

- you don't put much weight on superintelligence being a problem - therefore most of the bad outcome probability is from loss of control / gradual disempowerment / power concentration issues - so, those agendas are bad because they accelerate a world where companies have higher capability systems sooner

Are you trying to forever cap the capabilities of AI models? Is it a smooth transition thing?

It almost gave the impression that we should largely not make progress ever on alignment, which is hard to believe this is your take? It's a bit unclear to me what the ideal mainline future would look like to you. Do you never touch those agendas or make progress on alignment? You kind of have to at some point, no?