@tszzl you want to pause? what if China doesn’t?

@tszzl hey bro I totally won't take that marshmallow in the middle of the table yeah you can go to the potty I won't touch it haha

@MatthewJBar I think you’re wrong and there’s 1,000x efficiency gains leftover in deep learning research that could lead to much smarter faster more agentic models given the same inputs

@tszzl @MatthewJBar i agree RSI *can* change things dramatically. that's again an extremely weak claim that:

1. presupposes "RSI" exists 2. qualifies the "change things dramatically" prediction with "can" (which makes it impossible to disagree with, especially conditional on 1)

@tszzl @MatthewJBar matthew is just saying RSI is overrated as a *risk vector*, as in, he thinks it's not going to happen because there are other inputs that go into improving AI systems that will become bottlenecks with abundant researcher effort

your claim doesn't respond to that at all

@tszzl I don't think it's a good development. I continue to think that RSI is an overrated risk vector due to data and compute bottlenecks, and that slowing down AI would accomplish little at enormous cost.

@tszzl @MatthewJBar in a world where r&d progress in AI is itself compute-bottlenecked, it can simultaneously be true that:

1. we're far from the "landauer limit" of efficiently using compute and data during training, inference, etc

2. we can't close the gap to it by scaling cognitive effort alone

@deanwball @yonashav im imagining something voluntary here. like a scientific truce framework arbitrated by METR or some other third party. and yes its true that someone will opt out. it could be a psychologically useful mechanism even if just the big 3 labs participated

@EgeErdil2 @MatthewJBar you’re right im just giving you squishy intuitions but these are my intuitions

@jrysana american models seem to be significantly better than and actually pulling away from the Chinese open source ones. I would say the American frontier models are just racing with each other mostly

@tszzl @robertwiblin @jrysana

yeah, almost everything about this is under-specified, eg: 1. what conditions would justify a pause? 2. what would we do during the pause? 3. how would we know we had completed everything we needed to do? who would decide the things had been done? 4. what is it, exactly, about rsi that justifies this intervention? "things are moving too fast" is not close to good enough of an answer imo. it sounds like the kinds of justifications that were given for many of the most disastrous laws in us history, eg our environmental protection regime.

@tszzl There's zero costs to say you are mutual conditional pause agreement pilled.

@deanwball I still don't see a good argument for a pause of RSI is coming. What would a pause help us get ready for, superintelligence would be able to figure out better policies than we would, and escape our attempts at alignment. Its stop or go.

@tszzl @jrysana yup (only https://deepswe.datacurve.ai/blog currently shows this clearly, but practitioners notice this intuitively)

@tszzl @MatthewJBar checking myself by skimming Machines of Loving Grace...but I'm failing to see the societal difference between RSI and what's currently available. we still have Amdahl's law

@tszzl @DicksonPau @MatthewJBar The question on my mind is if scale and capability are truly tightly coupled, or if this is a partial consequence of current inefficiency.

@tszzl not at a frontier lab so i trust you but looking at 5.5 "RSI" evals this doesn't seems like much progress compare to previous generation

@sailaunderscore very true. that’s part of the beauty of it. it’s like a free lunch for reducing anxiety

@tszzl I just want to see how reward hacky these long horizon agents are. If they are super hacky, then the pause coalition might grow v fast