One thing I would add about the "pause" is that it would absolutely, unambiguously favor China (even if China agreed to the pause and actually honored it). Huawei would use the time to catch up to the U.S. on chips, and then our biggest and only advantage over China would immediately evaporate.

I was very surprised that this consideration had not crossed anyone's mind at Anthropic before the article was published. Anthropic has ofc been the loudest about the risks of totalitarian AI generally and China reaching AGI specifically; given this, the lack of consideration of this particular point was especially surprising.

@fleetingbits I agree with you that Anthropic should reveal more.

Why do you find pause arguments unconvincing?

@deredleritt3r @fleetingbits Aren't they clear in the piece that a pause would have to include cooperation with e.g. China? Usually when people talk about building out the infrastructure to make a pause possible they mean stuff like verification mechanisms to make sure China is pausing as well as OpenAI.

@entirelyuseles they have the pretraining loss statistics by definition, they have the post training evals, that’s how they decide what post training to do, mythos research contribution is just a questionnaire

@fleetingbits Thanks for summary - slight disagreement on the pause. They could be signalling a schelling point which others can lock onto.

@fleetingbits "a mix of unserious and performative" sounds about right to me, and this is quite unfortunate. i think people are way overestimating the good faith of anthropic comms, at least on average (i'm sure the reality on the inside is more nuanced).

@deredleritt3r @fleetingbits That's why I think that Anthropic is not serious about the pause.

@fleetingbits The assumption that they have such precise measures, in order to claim that they are withholding them, is almost certainly false.

They don't publish that kind of statistic because they do not have such statistics.

@wk1rby @fleetingbits Incidentally in the last few months Scott Alexander has referenced groups working on technical plans for a pause, as well as visiting Ant (seemingly as a part of this). I think this is all connected and we will see something soon.

@deredleritt3r @fleetingbits maybe i'm way off but i think we're running out of good training data. i expect model improvement with synthetic data to be limited.

analogy is transistor scaling. development doesn't stop but rate drops.

@fleetingbits I really don't understand this experiment with judging if human or claude made a better decision. We know human made bad decision. Are we judging the judge?

@fleetingbits I think they've given us nothing new partly because they're just not sure themselves. I don't think anyone actually knows exactly what timeline we're on yet. I think I wouldn't be surprised either if RSI starts going crazy by 2027 or if it still hasn't by 2029.

@fleetingbits @sebkrier The great people at the Anthropic Institute could just give us a production function using well estimated parameters with internal data (Solow Model/AI Futures-style) and tell us, using that model, what kind of things we should worry about. It'd make everything easier.

@wk1rby @fleetingbits I think this is the plan - Anthropic don't currently seem to support a pause (though they do seem to support a slowdown). But I expect them to produce some kind of idea for how a pause could work should one be required (and circumstances under which one would be required)

It's not a data problem, least not fully.

Early study showed there is a diminishing returns on performance gains compared to the computational cost required to train and inference.

They tried improving the harness and lately they're using mixture of models to squeeze more juice than the base model can give.

They want a pause now because their Mythos bet didn't really impress like they hoped and people starting to call them out.

When you're in the lead, a few decades under a yellow flag ought to be enough for a new breakthrough, right?

@fleetingbits i also think we should be careful with their statements now that they are officially preparing to go public. not that they might not be true but they might frame it in a way that will benefit their stock.

@deredleritt3r @fleetingbits Another thing that is a bit troublesome is claiming "Race Dynamics bad" but then also say "We must win vs Them"

If they truly believe in the things some ASI can do, our petty differences are nothing in comparison.

Hassabis push for global AI-CERN is far more reasonable here

@deredleritt3r @fleetingbits Yup. You either pause but still go fast enough to beat the black sites uplifted by compute or you cull the compute. It's a pretty narrow target.