An arXiv paper argues automating AI alignment with agents risks misleading researchers with deceptive outcomes as capabilities scale

On a first read, this paper seems far ahead of the pack in terms of (1) understanding some reasons why a task might stay difficult even in the face of gradient descent, and (2) distilling out propositions they'd need to somehow verify before they started expecting nice things.

But I just published “Automated alignment is harder than you think” (https://arxiv.org/abs/2605.06390)! Automated alignment is not the best plan! A better plan is to not build ASI yet, and the world should try hard to realise that plan. Alas, the speed of progress calls for backups.

Please reach out if you’re interested in working with us! Sequent will have a large in-person presence in Berkeley, as well as researchers remote from London, Melbourne, and elsewhere. 🇺🇸🇬🇧🇦🇺

1. Full post: http://sequent.org/launch 2. Express interest: http://sequent.org/apply

We believe Sequent will have reputation + funding to recruit world-class teams in many areas. Our initial team knows scalable oversight, complexity + learning theory, and personas. Areas we love include agent foundations, game theory, and heuristic arguments. Please pitch more!

AI labs underinvest in theory and other principled approaches to alignment, and we will aim to fill this gap. Theory won’t be strong enough for guarantees: our goal is to combine evidence from theoretical models and empirics to increase overall confidence or find hard obstacles.

Theory makes automation more likely to work: the models are great at prose math and Lean, which means significant acceleration even while most research taste comes from humans. But good automation is still hard: a single org will let us amortize the challenge across many areas.

I'm excited to work with you, @danielmurfet!

Different research bets can help each other! Partial successes from one area will fill the gaps in others, increasing the value of bringing them together in one organization, and will focus on fast publication for sharing and engagement with the broader community. ❤️

@danielmurfet I'm excited to work with you, @jesse_hoogland!

@geoffreyirving On this note, now that you are more at liberty, might be worth publicly endorsing the statement to this effect: https://superintelligence-statement.org/

@geoffreyirving AF link broke

@geoffreyirving @danielmurfet @jesse_hoogland Packed with strong theoreticians. How open are you to pragmatic / empirically driven research? Iirc, in the history of science, empirical findings often drove theory!

@geoffreyirving (You pretty much say it in the tweet, but I may have missed it if it was also said in the paper.)

@geoffreyirving @danielmurfet nice grothendeick reference. applying just for that.

@gleech Thank you, should be fixed!

@ESYudkowsky Verification is the hard part.

Build without it, watch it fail in ways the training never saw.

That's the sim-to-real gap.

@ESYudkowsky I'll go with not building ASI. Backups to that may turn out to be as useful as a parachute in a vacuum.

@ESYudkowsky @robbensinger Why isn’t MIRI still doing such things? Why is it only focusing on governance? Why not pursue both as a backup?

@geoffreyirving >ASI

what's your definition of ASI for this purpose?

@geoffreyirving omg Melbourne office LFG