AI Judge changed title after evaluation, original title: "UK AISI Chief Scientist Geoffrey Irving launches Sequent Research to focus on automated AI alignment"
The nonprofit plans to build automated alignment research tools
We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵
Users are excited about the Sequent Research nonprofit launch for aligning superintelligence because they anticipate collaborating on the project and following its progress.
this is my superbowl
We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵
Please reach out if you’re interested in working with us! Sequent will have a large in-person presence in Berkeley, as well as researchers remote from London, Melbourne, and elsewhere. 🇺🇸🇬🇧🇦🇺
1. Full post: http://sequent.org/launch 2. Express interest: http://sequent.org/apply
Sequent’s goal is to clear a higher bar:
1. We are aiming at higher confidence via a portfolio of theory and empirics bets (which could all fail!) 2. We’ll invest heavily in automation for fast progress 3. Theory boosts automation, via better filters for good research directions
Artificial superintelligence (ASI) may be developed in the next few years, and alignment is not on track! At a minimum, empirical research at AI labs is unlikely to deliver confidence, before training ASI, that alignment will go well.
AI labs underinvest in theory and other principled approaches to alignment, and we will aim to fill this gap. Theory won’t be strong enough for guarantees: our goal is to combine evidence from theoretical models and empirics to increase overall confidence or find hard obstacles.
Artificial superintelligence (ASI) may be developed in the next few years, and alignment is not on track! At a minimum, empirical research at AI labs is unlikely to deliver confidence, before training ASI, that alignment will go well.
Our full announcement post has more details. Please express interest if you’d like to join us!
1. Full post: http://sequent.org/launch 2. Express interest here: http://sequent.org/apply
We believe Sequent will have reputation + funding to recruit world-class teams in many areas. Our initial team knows scalable oversight, complexity + learning theory, and personas. Areas we love include agent foundations, game theory, and heuristic arguments. Please pitch more!
Theory makes automation more likely to work: the models are great at prose math and Lean, which means significant acceleration even while most research taste comes from humans. But good automation is still hard: a single org will let us amortize the challenge across many areas.
Different research bets can help each other! Partial successes from one area will fill the gaps in others, increasing the value of bringing them together in one organization, and will focus on fast publication for sharing and engagement with the broader community. ❤️
We believe Sequent will have reputation + funding to recruit world-class teams in many areas. Our initial team knows scalable oversight, complexity + learning theory, and personas. Areas we love include agent foundations, game theory, and heuristic arguments. Please pitch more!
Theory makes automation more likely to work: the models are great at prose math and Lean, which means significant acceleration even while most research taste comes from humans. But good automation is still hard: a single org will let us amortize the challenge across many areas.
But I just published “Automated alignment is harder than you think” (https://arxiv.org/abs/2605.06390)! Automated alignment is not the best plan! A better plan is to not build ASI yet, and the world should try hard to realise that plan. Alas, the speed of progress calls for backups.
I'm excited to work with you, @danielmurfet!
Timaeus was a beautiful dream, of using the Rising Sea of mathematics to turn the wheel of progress on the alignment problem. But we need the sea to rise faster. Time for some new axioms! 🧵
Our full announcement post has more details. Please express interest if you’d like to join us!
1. Full post: http://sequent.org/launch 2. Express interest here: http://sequent.org/apply
@danielmurfet I'm excited to work with you, @jesse_hoogland!
Timaeus is joining forces with @geoffreyirving and researchers from UK AISI to found Sequent Research. 1/9
We are passing through the valley of “technical slop” (wrong code, erroneous calculations) into the uncanny valley of “conceptual slop” where the models are right but pedestrian. A whole research field (in a datacenter?) can waste its time with wrong definitions and concepts.
If AGI is possible then automated alignment research is possible. The question is how early we can get it, and how we can tell the difference between apparent progress and real progress in a conceptually difficult and not-yet-formalised domain.
But we don’t need to fully automate research to get twenty years of progress in two: we “just” have to 10⨉ the rate. So the question is: can we leverage AI models + human conceptual and technical direction, to vastly increase the rate?
A combination of human direction, autoformalisation in Lean and automated experimentation on formalised predictions is already today unlocking a rising tide of (modest, but real) progress on basic aspects of theoretically driven research agendas. This will rise with the models.
This is why a strong human component is still necessary in alignment research at Sequent. Maybe that’s you. You should consider dropping what you’re doing and helping. A lot of other theoretical and empirical research can be left to the ASI, but alignment can’t (responsibly).
Very excited to be a part of this!
We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵
This is the premise of Sequent. We believe theory- and understanding-based approaches are well-suited to this approach. A theorem is worth a thousand experiments, and even a pseudo-proof is worth hundreds. We worry this kind of work will not take place by default in AI labs.
We developed a new interpretability stack (“spectroscopy”) and scaled it to models with billions of parameters. These techniques discover rich internal structure that is competitive with SAEs but based on a different foundation (weights rather than activations). 3/9
AI Judge changed title after evaluation, original title: "UK AISI Chief Scientist Geoffrey Irving launches Sequent Research to focus on automated AI alignment"
The nonprofit plans to build automated alignment research tools
We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵
