Gary Marcus warns of 'AI successionists' who argue that AI should replace humanity as a natural evolutionary step

I reported this deep dive for @voxdotcom because I think more folks really need to be aware of this — here’s a gift link:

https://www.vox.com/future-perfect/489976/ai-successionism-transhumanism-posthumanism?view_token=eyJhbGciOiJIUzI1NiJ9.eyJpZCI6IlZCM29tNWp0b3giLCJwIjoiL2Z1dHVyZS1wZXJmZWN0LzQ4OTk3Ni9haS1zdWNjZXNzaW9uaXNtLXRyYW5zaHVtYW5pc20tcG9zdGh1bWFuaXNtIiwiZXhwIjoxNzgyNzMxMjM2LCJpYXQiOjE3ODE1MjE2MzZ9.sQ0wZv7kqdR8v5IxDIT2mbBukOH2NkxhLLD1mbzU4m8&utm_medium=gift-link

2/n

@SigalSamuel Most everyone wants someone other than the current generations of humans to replace them.

@voxdotcom @seanilling @danfaggella And the thing is, (A) and (B) are perfectly reasonable! But (C) is very questionable. Yet many smart, well-meaning people get snowed into it because they don’t have a better mental model. I think that’s because classical humanism has failed us. So, we need to try again. 12/n

@voxdotcom Looking for a podcast for your commute? I spoke to @seanilling about the AI successionists on this episode of The Gray Area — it was a great convo where we were able to get into some of the nuances:

https://open.spotify.com/episode/7AWK7Z1Daywj7OaX6qYOK5

@voxdotcom @seanilling @danfaggella These are people who earnestly believe they’re more enlightened for saying (A) human exceptionalism is wrong, (B) it’s naïve to think Homo Sapiens will stay the same forever, (C) we should be loyal not to Homo Sapiens but only to life/intelligence/consciousness itself. 11/n

@voxdotcom @seanilling AI successionists believe machines could become our moral superiors, so they argue it’s actually wrong to try to keep them down. Instead, we should welcome artificial intelligence as a successor to humanity and hand over the world to it. Even if that means we go extinct. 4/n

@voxdotcom @seanilling In the course of my reporting, some AI successionists, like Richard Sutton, said things that really worried me: 5/n

@voxdotcom @seanilling @danfaggella To be clear, not everyone in this group is an AI accelerationist. @danfaggella is deeply invested in the idea of a “Worthy Successor” (“successor” as in “succession”), but, to his credit, he has written that it would be really dumb to race toward AGI. I agree! 8/n

@voxdotcom @seanilling Most successionists are more tight-lipped about their taboo views. They wield a lot of influence, though, from their positions within the top AI companies and within think tanks that directly shape US government policy on AI. 6/n

@voxdotcom @seanilling A few months ago I went to an invite-only symposium in NYC dedicated to the idea of creating a “Worthy Successor.” Here’s how organizer @danfaggella defines it: 7/n

@voxdotcom @seanilling @danfaggella That’s why in my article I don’t only try to reject a view; I try to build up a countervailing positive vision — a humanism fit for the 21st century. In a nutshell:

13/n

@voxdotcom @seanilling @danfaggella These are just opening moves. But for anyone who feels “surely there must be a solid position that’s neither ‘it’s fine if we build super-good AI & humans go extinct’ NOR ‘humans are the good-est & must stay the same forever’” — you’re right. I plan to keep fleshing it out. /FIN

@voxdotcom @seanilling @danfaggella But many successionists are accelerationists, including many of the most powerful & high-profile ones. Like family members, they don't have every feature in common but they do share a “family resemblance” (a la Wittgenstein). Successionism & accelerationism often travel together.

@voxdotcom @seanilling @danfaggella So what are you supposed to make of these people? The point is not “these must be straight-up villains, how else could they arrive at such outlandish views”! The point is... 10/n

@SigalSamuel @voxdotcom You’re a very good writer, & I’m glad that you’re exposing the cult of effective altruism to a broader audience. These are folks with tremendous wealth & power, & society needs to understand this better. I do wish you would’ve cited folks like @xriskology & @timnitGebru

@neilturkewitz @SigalSamuel @voxdotcom @timnitGebru My guess is that FP writers have been instructed not to cite me. It was amusing a few years back to see x-risk scholars writing about "agential risks" while doing acrobatics to avoid citing my work on the topic (which they were clearly drawing from). Citational politics!

@SigalSamuel I don't believe we are on the verge of a new posthuman intelligence and also don't believe in linear scale of "intelligence" such that we should always be happy to have more intelligent "replace" less, but disagree w idea we should just toss out desire for universalist morality

@SigalSamuel @voxdotcom @seanilling @danfaggella Humanity is a region of mind space, not a particular animal host.

@GaryMarcus @xriskology I've been warning about this too for 2 to 3 years and I have gone over the receipts.

It's been around for a long time. Decades.

@SigalSamuel If you casually reject universalism in favor of this sort of particularism, what stops you from applying the same argument to some groups of humans vs. others, eg "I'm a member of this culture/race so naturally I'll do whatever makes life best for 'us' even at expense of others?"