Anthropic releases paper arguing United States and allies must maintain frontier AI lead over China through continued competitive steps
Nate Soares criticizes Anthropic for encouraging AI race without coordination
Do you disagree with the article?
As far as I can tell, it's making the correct point that America shouldn't be leaking chips to China. Fighting this smuggling is the correct move even within (especially within!) a high-pdoom worldview.
If negotiation is possible, China is more likely to negotiate when they're losing (or when we have a carrot to offer them, in the form of chips that we're not giving for free).
If negotiation is impossible, then it's better to have all the AI development concentrated in one country. That country then at least has the option to pause/slowdown AI for however long it takes the other countries to catch up, even if it can't do so permanently. Or it can regulate AI without having to worry about losing the race. I tried to make this case at https://www.astralcodexten.com/p/why-ai-safety-wont-make-america-lose , which I think makes the same anti-compute proliferation arguments Anthropic is making on their blog post, from a specifically safety-oriented perspective.
I think attacking Anthropic for fighting compute proliferation is a net negative even within what I think is your own world-model. Any successful slowdown will come from a hundred small things going right beforehand that convince everyone it's in their best interest (like the US cracking down on compute leaking to other countries). If you attack every attempt to make small things go right because it's not the big thing you want, you're decreasing the chance of ever getting the big thing.
Nate Soares ⏹️@So8resAnthropic encourages racing without even acknowledging the possibility of global coordination (below). They hire top scientists (Karpathy) to work on the most dangerous tech (recursive self-improvement). This is not "good guys" behavior.
There is something here:
Nate Soares ⏹️@So8resAnthropic encourages racing without even acknowledging the possibility of global coordination (below). They hire top scientists (Karpathy) to work on the most dangerous tech (recursive self-improvement). This is not "good guys" behavior.
@So8res I think i’d say that pushes forward the frontier, rather than most dangerous tech. I think RSI is justifiable if everyone is doing it, but bad if you’re pushing the frontier.
Nate Soares ⏹️@So8resAnthropic encourages racing without even acknowledging the possibility of global coordination (below). They hire top scientists (Karpathy) to work on the most dangerous tech (recursive self-improvement). This is not "good guys" behavior.
@So8res I agree.
Nate Soares ⏹️@So8res@NathanpmYoung "We must do this thing that horribly endangers you bc if we don't the next guy will do it even more dangerously" is a possible justification, but it comes with a solemn responsibility to do everything in your power to help Earth find a third alternative. Anthropic fails that test
They pretend like they're a "safe" AI company; a "good" company; but insofar as they're racing to enrich themselves by endangering your life *without* also begging the world to please find some other way, they're betraying that mantle.
Nate Soares ⏹️@So8resAnthropic encourages racing without even acknowledging the possibility of global coordination (below). They hire top scientists (Karpathy) to work on the most dangerous tech (recursive self-improvement). This is not "good guys" behavior.
@NathanpmYoung "We must do this thing that horribly endangers you bc if we don't the next guy will do it even more dangerously" is a possible justification, but it comes with a solemn responsibility to do everything in your power to help Earth find a third alternative. Anthropic fails that test
Nathan is in Berkeley 🔎@NathanpmYoung@So8res I think i’d say that pushes forward the frontier, rather than most dangerous tech. I think RSI is justifiable if everyone is doing it, but bad if you’re pushing the frontier.
I'm not trying to highlight inaccuracies; I'm trying to highlight a missing mood.
I think any attempt to say "we're forced into doing this horribly reckless thing that might kill you and your family, because if we don't then the next guy will do it even more dangerously" comes with a solemn responsibility to do everything in your power to help the world find some third alternative. I think Anthropic fails this test pretty badly, e.g. as evidenced here: https://x.com/AnthropicAI/status/2031674089035714578 and as Rob documented a bit here: https://x.com/robbensinger/status/2027848301236912525.
Over the last few months, reporters have asked me some variant of "but what about Anthropic? Aren't they a safe company? Do you hope that they win, as the good guys?" a handful of times. This causes me to think that a bunch of people are moved by the "we're the good guys" act.
I think it matters, strategically, as to whether all the world needs right now is the Right Company to Win, or whether we need something more like a global shutdown. So I think it's important to correct what seems to me like a common misconception around anthropic. I also think a lot of locals are loathe to criticize anthropic for one reason or another (they work there; their friends work there; they think they're better than OpenAI; ...). Thus, it looks to me like I can probably make a positive difference by highlighting ways that Anthropic is (afaict) dramatically failing to carry the "safe/good AI company" mantle.
(I tend to think it's even more important to communicate how even a company that *was* living up to the mantle still wouldn't have much of a chance, and how the real solution is an international shutdown. But I don't have to pick just one. When current events evidence some of the difference between the niche Anthropic pretends to occupy and the niche Anthropic actually occupies, I try to take those opportunities.)
Scott Alexander@slatestarcodexDo you disagree with the article? As far as I can tell, it's making the correct point that America shouldn't be leaking chips to China. Fighting this smuggling is the correct move even within (especially within!) a high-pdoom worldview. If negotiation is possible, China is more likely to negotiate when they're losing (or when we have a carrot to offer them, in the form of chips that we're not giving for free). If negotiation is impossible, then it's better to have all the AI development concentrated in one country. That country then at least has the option to pause/slowdown AI for however long it takes the other countries to catch up, even if it can't do so permanently. Or it can regulate AI without having to worry about losing the race. I tried to make this case at https://www.astralcodexten.com/p/why-ai-safety-wont-make-america-lose , which I think makes the same anti-compute proliferation arguments Anthropic is making on their blog post, from a specifically safety-oriented perspective. I think attacking Anthropic for fighting compute proliferation is a net negative even within what I think is your own world-model. Any successful slowdown will come from a hundred small things going right beforehand that convince everyone it's in their best interest (like the US cracking down on compute leaking to other countries). If you attack every attempt to make small things go right because it's not the big thing you want, you're decreasing the chance of ever getting the big thing.