Anthropic publishes paper outlining scenarios for the US and democratic allies to maintain frontier AI lead over China, drawing criticism for limited global coordination discussion

VIEWS21KLIKES224REPLIES5

Disappointed to see Anthropic’s US/China scenarios have essentially no mention of a coordinated slowdown.

“US/China slowdown is obviously infeasible” attitude is self-fulfilling. (This piece doesn’t explicitly argue it, but its focus is telling and Dario has explicitly said it.)

Anthropic@AnthropicAI

We've published a paper that explains our views on AI competition between the US and China.

The US and democratic allies hold the lead in frontier AI today. Read more on what it’ll take to keep that lead: https://www.anthropic.com/research/2028-ai-leadership

40d21K22416

BOOKMARKS17

Nate Soares ⏹️@So8res

I'm not trying to highlight inaccuracies; I'm trying to highlight a missing mood.

I think any attempt to say "we're forced into doing this horribly reckless thing that might kill you and your family, because if we don't then the next guy will do it even more dangerously" comes with a solemn responsibility to do everything in your power to help the world find some third alternative. I think Anthropic fails this test pretty badly, e.g. as evidenced here: https://x.com/AnthropicAI/status/2031674089035714578 and as Rob documented a bit here: https://x.com/robbensinger/status/2027848301236912525.

Over the last few months, reporters have asked me some variant of "but what about Anthropic? Aren't they a safe company? Do you hope that they win, as the good guys?" a handful of times. This causes me to think that a bunch of people are moved by the "we're the good guys" act.

I think it matters, strategically, as to whether all the world needs right now is the Right Company to Win, or whether we need something more like a global shutdown. So I think it's important to correct what seems to me like a common misconception around anthropic. I also think a lot of locals are loathe to criticize anthropic for one reason or another (they work there; their friends work there; they think they're better than OpenAI; ...). Thus, it looks to me like I can probably make a positive difference by highlighting ways that Anthropic is (afaict) dramatically failing to carry the "safe/good AI company" mantle.

(I tend to think it's even more important to communicate how even a company that *was* living up to the mantle still wouldn't have much of a chance, and how the real solution is an international shutdown. But I don't have to pick just one. When current events evidence some of the difference between the niche Anthropic pretends to occupy and the niche Anthropic actually occupies, I try to take those opportunities.)

40d5.6K14317

RETWEETS41

Nate Soares ⏹️@So8res

Anthropic encourages racing without even acknowledging the possibility of global coordination (below). They hire top scientists (Karpathy) to work on the most dangerous tech (recursive self-improvement). This is not "good guys" behavior.

Anthropic@AnthropicAI

We've published a paper that explains our views on AI competition between the US and China.

The US and democratic allies hold the lead in frontier AI today. Read more on what it’ll take to keep that lead: https://www.anthropic.com/research/2028-ai-leadership

40d81.1K528123

Scott Alexander@slatestarcodex

@davidmanheim @So8res In a world where teachers are constantly having unprotected sex, a policy of condemning any statement by a teacher that they will start wearing a condom is making things worse.

39d2.2K412

Nate Soares ⏹️@So8res

They pretend like they're a "safe" AI company; a "good" company; but insofar as they're racing to enrich themselves by endangering your life *without* also begging the world to please find some other way, they're betraying that mantle.

40d736283

Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTex

@slatestarcodex @So8res Scott, one question. Have you considered a scenario where you "win the AI race" but it doesn't actually translate into any victory over Chyna, and instead you've just permanently pissed them off with this decapitation strike scheming, and they have like 70% of world industry.

Scott Alexander@slatestarcodex

Do you disagree with the article?

As far as I can tell, it's making the correct point that America shouldn't be leaking chips to China. Fighting this smuggling is the correct move even within (especially within!) a high-pdoom worldview.

If negotiation is possible, China is more likely to negotiate when they're losing (or when we have a carrot to offer them, in the form of chips that we're not giving for free).

If negotiation is impossible, then it's better to have all the AI development concentrated in one country. That country then at least has the option to pause/slowdown AI for however long it takes the other countries to catch up, even if it can't do so permanently. Or it can regulate AI without having to worry about losing the race. I tried to make this case at https://www.astralcodexten.com/p/why-ai-safety-wont-make-america-lose , which I think makes the same anti-compute proliferation arguments Anthropic is making on their blog post, from a specifically safety-oriented perspective.

I think attacking Anthropic for fighting compute proliferation is a net negative even within what I think is your own world-model. Any successful slowdown will come from a hundred small things going right beforehand that convince everyone it's in their best interest (like the US cracking down on compute leaking to other countries). If you attack every attempt to make small things go right because it's not the big thing you want, you're decreasing the chance of ever getting the big thing.

40d1K312

Nate Soares ⏹️@So8res

@NathanpmYoung "We must do this thing that horribly endangers you bc if we don't the next guy will do it even more dangerously" is a possible justification, but it comes with a solemn responsibility to do everything in your power to help Earth find a third alternative. Anthropic fails that test

40d367222

Nate Soares ⏹️@So8res

Attempting to spell out the connection between my reply and my understanding of Scott's point even more explicitly:

1. I agree that America having an obvious lead makes it easier to bring other countries (e.g. China) to the negotiating table. 2. I agree it's probably bad for America to ship advanced AI chips everywhere (with a caveat that I *also* think it's bad for American policy to signal "this is very valuable tech that everyone should race towards" as opposed to "this is very dangerous tech that none of us should be pursuing", and various chip controls can have a bad version of the former effect). 3. I agree it's mostly good for Anthropic to fight compute proliferation (modulo the caveat above). 4. I nevertheless think that the article is valid evidence of Anthropic failing to to live up to the "do everything in your power to help the world find a third alternative" responsibility that they adopt by racing as hard as they can towards, e.g., recursive self-improvement.

To spell out why I think it's valid evidence of a missing mood: I think that if the company were actually trying hard to achieve an international slowdown, the article would sound pretty different, and would (e.g.) mention that the ideal is a global slowdown. AFAICT the closest they get to that in the article is saying "Anthropic supports international AI safety dialogue with AI experts in China, when possible." which is *maybe* a dogwhistle for "actually this AI race is ridiculous and the whole world should be slowing down". But the dynamic where Anthropic communications rarely (if ever) come right out and *say* that (and perhaps instead try to assuage locals with dogwhistles) is the very dynamic I'm trying to draw attention to. This isn't what it looks like to try hard to find a third path! A banner on your website and getting down on your knees in front of congress to beg them to form a treaty would be closer! There's a big missing mood in their comms.

40d74737

Scott Alexander@slatestarcodex

I agree with the missing mood, I just don't like the idea of criticizing a company for doing an object-level good thing because the mood is wrong. They wrote their piece on this topic in approximately the same register I wrote mine. I wrote mine that way because I thought this is a rare chance to get national security hawks and AI safety adovcates on the same page, and I didn't want to lose my national security hawk readers.

39d78214

Oliver Habryka@ohabryka

I don’t think the articles most important point is that it’s about not smuggling chips to China.

The article’s most important point is that America should aim for a 24 month lead within 24 months, which basically precludes any interest in a mutual slowdown or pause.

It also directly threatens the rest of the world that America should should use AI technologies offensively or as an active threat in negotiations, which of course will predictably produce an arms race.

Yes, probably someone at Anthropic wrote this article with an aim to push the USG into doing more chip controls, but in the process of doing that they described a catastrophic AI policy that extends substantially beyond that.

40d433141

Nate Soares ⏹️@So8res

@slatestarcodex @davidmanheim If many folk say "But isn't Mr. Henderson one of the good teachers? Shouldn't he get lots of power?", I think it's fair game to occasionally say "he's still one of the baddies; you can tell this by how he talks about using condoms", even if you appreciate him using protection.

39d29321

Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTex

@slatestarcodex @So8res Sentiments aside, you forever lose IC market foothold in China and its economic sphere which will likely expand. I get that you're a man of letters but you probably can comprehend how *puny* American industry is, and how Wunderwaffe-like these dreams of AI leapfrogging are

Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTex

@slatestarcodex @So8res Scott, one question. Have you considered a scenario where you "win the AI race" but it doesn't actually translate into any victory over Chyna, and instead you've just permanently pissed them off with this decapitation strike scheming, and they have like 70% of world industry.

40d558150

Holly ⏸️ Elmore@ilex_ulmus

@slatestarcodex @So8res They aren’t “fighting compute proliferation”. They are using the language of the Cold War to 1) get more compute for themselves, 2) maintain China as a foil that justifies their own unsafe rapid development. They aren’t trying to stop compute proliferation to all the race leaders

40d23312

Scott Alexander@slatestarcodex

Do you disagree with the article?

As far as I can tell, it's making the correct point that America shouldn't be leaking chips to China. Fighting this smuggling is the correct move even within (especially within!) a high-pdoom worldview.

If negotiation is possible, China is more likely to negotiate when they're losing (or when we have a carrot to offer them, in the form of chips that we're not giving for free).

If negotiation is impossible, then it's better to have all the AI development concentrated in one country. That country then at least has the option to pause/slowdown AI for however long it takes the other countries to catch up, even if it can't do so permanently. Or it can regulate AI without having to worry about losing the race. I tried to make this case at https://www.astralcodexten.com/p/why-ai-safety-wont-make-america-lose , which I think makes the same anti-compute proliferation arguments Anthropic is making on their blog post, from a specifically safety-oriented perspective.

I think attacking Anthropic for fighting compute proliferation is a net negative even within what I think is your own world-model. Any successful slowdown will come from a hundred small things going right beforehand that convince everyone it's in their best interest (like the US cracking down on compute leaking to other countries). If you attack every attempt to make small things go right because it's not the big thing you want, you're decreasing the chance of ever getting the big thing.

40d24951

Campbell@abcampbell

@So8res @slatestarcodex I don’t think you are actually responding to his point.

40d7396

Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTex

@slatestarcodex @So8res puny in relative terms ofc the US is a big country, inhabited by many Americans etc etc

Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTex

@slatestarcodex @So8res Sentiments aside, you forever lose IC market foothold in China and its economic sphere which will likely expand. I get that you're a man of letters but you probably can comprehend how *puny* American industry is, and how Wunderwaffe-like these dreams of AI leapfrogging are

40d500130

Nathan is in Berkeley 🔎@NathanpmYoung

@So8res I think i’d say that pushes forward the frontier, rather than most dangerous tech. I think RSI is justifiable if everyone is doing it, but bad if you’re pushing the frontier.

40d4414

Campbell@abcampbell

@So8res @slatestarcodex In spite of your faux concession and attempt at reframe, you still aren't engaging Scott’s point.

40d3202

Scott Alexander@slatestarcodex

@aiamblichus @So8res I think a good analogy here is nuclear weapons. I also don't like Trump and Hegseth, but I'd rather have the US have nuclear hegemony than proliferate nukes to random third-party countries and US rivals.

39d1092

Afflatus Solis@AfflatusSolis

@slatestarcodex @So8res Why would they negotiate when they're losing instead of...say destroying American AI companies or starting a war or racing ahead on reckless things or invading Taiwan and destroying the compute for everyone?

40d9321