The group will train alignment specialists from non-AI fields.
Read the blog post. Most optimistic I’ve been about the alignment community and for advancing alignment research in a while.
We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵
@johnschulman2 debate as a scalable oversight mechanism was genuinely ahead of its time
curious how Sequent approaches the part where debate breaks down if both sides are misaligned
I remember talking to some authors at a poster session (Neurips?) that even for simple examples (eg whats the nearest restaurant) models can design extremely complex hard to falsify counterarguments and debates get extremely verbose very fast. Also: You get a lot of the bot replies with these weird anime characters, maybe just block them, waste of time to read them for the people following you.
Looking forward to what comes out of Geoffrey's new alignment org. His 2018 paper on AI safety via debate is one of my all-time favorites: an elegant framing of the scalable oversight problem, way ahead of its time.
We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵
"We will happily train people"
Geoffrey Irving says Sequent will recruit experts from many fields and train them in alignment rather than only hiring existing AI safety insiders.
"we will happily train people who are experts in various fields in alignment"
"You should not only reach out to us if you know AI safety well."
"just in combination with the machines."
@geoffreyirving
"the question is just how much?"
Daniel Murfet says Sequent is not trying to recursively improve the whole model stack, and current alignment theory may still be low-complexity enough.
"we're more picking up the what's on the table to be applied to alignment"
"all the concepts that are around in alignment right now are very, like, low level"
"the question is just how much?"
@danielmurfet
@johnschulman2 the debate paper was wild to reread last year, holds up way better than most from that era
curious how the public tracking will work vs the core research
Follow @AI_in_the_AM for the daily rundown!
Full video: https://briefing.ai-in-the-am.com/p/aiam-fable-ai-safety-and-julius-june-10-2026
Follow @AI_in_the_AM for the daily rundown!
Full video: https://briefing.ai-in-the-am.com/p/aiam-fable-ai-safety-and-julius-june-10-2026
@johnschulman2 debate is that paper people cite but somehow never agree on what it actually proved
maybe thats the point though
@johnschulman2 hope they keep debate as a core framework instead of it becoming another vac benchmark chase
papers that shape a field deserve better than that
The group will train alignment specialists from non-AI fields.
Read the blog post. Most optimistic I’ve been about the alignment community and for advancing alignment research in a while.
We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵
