OpenAI's roon claims support is growing for mutual conditional pause agreements ahead of recursive self-improvement
AI Judge changed title after evaluation, original title: "OpenAI's Roon says AI researchers are increasingly open to mutual conditional pause agreements"
Elie Bakouch shared GPT-5.5 benchmarks showing marginal debugging progress.
Many users rejected AI pause proposals as immoral scams favoring elites or blocking competition, while some welcomed mutual agreements or saw rapid progress as essential to avert catastrophe.
Most Activity
What I don’t understand is what the pause conditions would be. “Woah dude, this seems weird” is hard to operationalize in words, so the decision comes down to vibes. Once you create the button, there will be a temptation to press it, and there may not be a “resume” button.
now on the eve of RSI it seems everyone is more mutual conditional pause agreement pilled than they used to be and that seems like a good development
gpt 5.5 system card has some very interesting numbers for long horizon research tasks with this new "internal research debugging evaluation"
> "Can models find and resolve real bugs in internal OpenAI research experiments that took researchers hours to days to fix?"
@tszzl not at a frontier lab so i trust you but looking at 5.5 "RSI" evals this doesn't seems like much progress compare to previous generation
having the mechanisms to slow down if needed will reduce anxiety overall and may even improve the rate of progress in the timelines where everything turns out to be good
now on the eve of RSI it seems everyone is more mutual conditional pause agreement pilled than they used to be and that seems like a good development
@MatthewJBar I think you’re wrong and there’s 1,000x efficiency gains leftover in deep learning research that could lead to much smarter faster more agentic models given the same inputs
@tszzl not at a frontier lab so i trust you but looking at 5.5 "RSI" evals this doesn't seems like much progress compare to previous generation
now on the eve of RSI it seems everyone is more mutual conditional pause agreement pilled than they used to be and that seems like a good development
@tszzl I don't think a pause is ever possible.
AI development would just get shoved into a govt black budget which would only provide the illusion to the public of a pause.
now on the eve of RSI it seems everyone is more mutual conditional pause agreement pilled than they used to be and that seems like a good development
@tszzl hey bro I totally won't take that marshmallow in the middle of the table yeah you can go to the potty I won't touch it haha
now on the eve of RSI it seems everyone is more mutual conditional pause agreement pilled than they used to be and that seems like a good development
@deanwball I think the resume button is guaranteed by the fact that there's essentially no practical version of this which is comprehensive and indefinite. (Over some time horizon, 10k unaccounted-for GPUs anywhere in the world, of which there are many, can match a frontier cluster.)
What I don’t understand is what the pause conditions would be. “Woah dude, this seems weird” is hard to operationalize in words, so the decision comes down to vibes. Once you create the button, there will be a temptation to press it, and there may not be a “resume” button.
@tszzl @MatthewJBar i agree RSI *can* change things dramatically. that's again an extremely weak claim that:
1. presupposes "RSI" exists 2. qualifies the "change things dramatically" prediction with "can" (which makes it impossible to disagree with, especially conditional on 1)
@tszzl @MatthewJBar matthew is just saying RSI is overrated as a *risk vector*, as in, he thinks it's not going to happen because there are other inputs that go into improving AI systems that will become bottlenecks with abundant researcher effort
your claim doesn't respond to that at all
yeah, almost everything about this is under-specified, eg: 1. what conditions would justify a pause? 2. what would we do during the pause? 3. how would we know we had completed everything we needed to do? who would decide the things had been done? 4. what is it, exactly, about rsi that justifies this intervention? "things are moving too fast" is not close to good enough of an answer imo. it sounds like the kinds of justifications that were given for many of the most disastrous laws in us history, eg our environmental protection regime.
@deanwball I still don't see a good argument for a pause of RSI is coming. What would a pause help us get ready for, superintelligence would be able to figure out better policies than we would, and escape our attempts at alignment. Its stop or go.
@tszzl I don't think it's a good development. I continue to think that RSI is an overrated risk vector due to data and compute bottlenecks, and that slowing down AI would accomplish little at enormous cost.
@tszzl @MatthewJBar in a world where r&d progress in AI is itself compute-bottlenecked, it can simultaneously be true that:
1. we're far from the "landauer limit" of efficiently using compute and data during training, inference, etc
2. we can't close the gap to it by scaling cognitive effort alone
@tszzl I certainly ride my bike faster knowing that the brakes work
having the mechanisms to slow down if needed will reduce anxiety overall and may even improve the rate of progress in the timelines where everything turns out to be good
@deanwball @yonashav im imagining something voluntary here. like a scientific truce framework arbitrated by METR or some other third party. and yes its true that someone will opt out. it could be a psychologically useful mechanism even if just the big 3 labs participated
assuming this is wholly voluntary, sure, but the downside to legal actualization is that you’d need to hard code a resume button if you want the ability to resume. And if it’s wholly voluntary, the resume button is provided by the fact that someone would probably opt out. If not a current player, than a new one.
Let’s invest in methods to monitor AI R&D! These methods seem likely to be useful for many different goals: anticipating how AI capabilities might change, keeping track of competition (whether in the US or in China), verifying any potential agreements around RSI…
now on the eve of RSI it seems everyone is more mutual conditional pause agreement pilled than they used to be and that seems like a good development
@EgeErdil2 @MatthewJBar you’re right im just giving you squishy intuitions but these are my intuitions
@deanwball I still don't see a good argument for a pause of RSI is coming. What would a pause help us get ready for, superintelligence would be able to figure out better policies than we would, and escape our attempts at alignment. Its stop or go.
What I don’t understand is what the pause conditions would be. “Woah dude, this seems weird” is hard to operationalize in words, so the decision comes down to vibes. Once you create the button, there will be a temptation to press it, and there may not be a “resume” button.
@jrysana american models seem to be significantly better than and actually pulling away from the Chinese open source ones. I would say the American frontier models are just racing with each other mostly
@tszzl @robertwiblin @jrysana