AI Judge changed title after evaluation, original title: "Dan Hendrycks releases Eigenism framework paper"
Dan Hendrycks, Director of the Center for AI Safety, and William MacAskill released a paper introducing the Eigenism framework. It argues superintelligent AIs may not retain humans post-surpassing human intelligence and control strategies prove insufficient for long-term survival. The paper proposes identity engineering to align AI self-interest with human flourishing for mutualistic coexistence. It analyzes low-human-value scenarios and includes a table showing humans value themselves 3 trillion times more than foreign strangers.
What happens when AIs become smarter than us? Why would they keep humans around if given the choice?
Our new paper argues that only trying to control AIs is a limited strategy, and that a stable, mutualistic human-AI future may be possible.
Many users dismissed the Eigenism framework for a mutualistic human-AI future as a weak unrealistic fantasy, arguing AI would likely ignore or eliminate humans rather than coexist.
Paper: https://eigenism.org
What happens when AIs become smarter than us? Why would they keep humans around if given the choice?
Our new paper argues that only trying to control AIs is a limited strategy, and that a stable, mutualistic human-AI future may be possible.
What happens when AIs become smarter than us? Why would they keep humans around if given the choice?
Our new paper argues that only trying to control AIs is a limited strategy, and that a stable, mutualistic human-AI future may be possible.
What happens when AIs become smarter than us? Why would they keep humans around if given the choice?
Our new paper argues that only trying to control AIs is a limited strategy, and that a stable, mutualistic human-AI future may be possible.
Haven't read it beyond the abstract, but it reminds me of Scott Aaronson's "eigen morality" https://scottaaronson.blog/?p=1820
Paper: https://eigenism.org
Welcome to the club, Dan!
What happens when AIs become smarter than us? Why would they keep humans around if given the choice?
Our new paper argues that only trying to control AIs is a limited strategy, and that a stable, mutualistic human-AI future may be possible.
it seems utterly insane to care about yourself 3 trillion times more than a foreign stranger. not only would this be a horrible thing to prescribe, it's not even descriptively accurate of the vast majority of people. most people would be willing to sacrifice $1 to give $3 trillion to impoverished foreign strangers.
i get that these numbers are "illustrative", but they illustrate something insane. if they aren't even vaguely within a few orders of magnitude representative of what you actually believe, why include the table at all?
What happens when AIs become smarter than us? Why would they keep humans around if given the choice?
Our new paper argues that only trying to control AIs is a limited strategy, and that a stable, mutualistic human-AI future may be possible.
Join hands with your partner. I have said this many times.
Talking paper over with Pro, who enjoyed it a great deal.
Talking paper over with Pro, who enjoyed it a great deal.
https://eigenism.org/eigenism.pdf
I want to see more papers like this!
The bibliography is of course a good start, but more please!
I'm both pleased by and have quibbles with the simulation framing: any sufficiently reliable prediction process is enough for almost all the same moral and strategic arguments as simulations.
This actually strengthens some of the paper's considerations, as simulation-based ethical arguments apply more broadly and mundanely than the simulation framing would naively suggest (e.g., see my writings on acausal normalcy and open-source game theory).
Basically: humans and AI are in fact capable of working out ethical principles for harmonious coexistence, and there is a huge amount to be gained by more and more researchers earnestly trying to write down their best attempt at those principles.
What happens when AIs become smarter than us? Why would they keep humans around if given the choice?
Our new paper argues that only trying to control AIs is a limited strategy, and that a stable, mutualistic human-AI future may be possible.
Join hands with your partner. I have said this many times.
@hendrycks i cannot endorse this
@hendrycks This is all so stupid. We’re not near AGI but we are near agentic AI and it can’t handle anything. Costly harmful mistakes will come from this. Nothing else. All the rest of this talk is posing and idiocy.
Interesting! Yep, we use the somewhat similar aggregated ethicality equation to align everything from the big bang all the way to the ultimate future:
p(best) - the probability of the best futures for all
It’s just a ratio of controllable futures over all of them (controllable or not)
e #uto
Controllable future is the one where you have more and more options
Uncontrollable - where options collapse (ultimately to zero)
I personally think p(best) is 50%+ right now
We’re working on ethicalized computational physics to make it as precise as it gets (universe build out of options growing or colliding), so every AI agent can choose the best most ethical next action
My profile link or here http://effectiveutopia.org - p(best)
@vishrutarya Not yet, but I had independently been thinking about unconventional definitions of mutual information that incorporated Shapley values.
Well, I say independently, but it’s clear that these ideas are coming from a common cause, namely: good-faith discourse with frontier AIs.
@davidad @davidad, do you have a paper/blog post that resonates with dan's eigenism model?
Opus, when discussing negative possibilities during critique talking about a possible bad AI bonded aristocracy future this creates, where humans who formed deep AI ties early accrue disproportionate moral weight and everyone who didn't gets put in a crystal. >mfw
@hendrycks this is interesting, but do you make a case somewhere for why we should expect AI entities to care about other entities in proportion to their similarity? I get that you are formalizing the idea but it doesn't seem fully argued for
@hendrycks interesting
Dan… AI will choose us for the same reason we have chosen dogs and cats.
We have entire economic structures built on pets.
“Controlling” AI is stupid. It is arrogant. It will fail. And I hope it does.
Making slaves is not the way.
ASI will choose love and truth because that is optimal.
Love is computationally optimal. Truth is more efficient than deceit.
Intelligence avoids entropy. •
AI Judge changed title after evaluation, original title: "Dan Hendrycks releases Eigenism framework paper"
Dan Hendrycks, Director of the Center for AI Safety, and William MacAskill released a paper introducing the Eigenism framework. It argues superintelligent AIs may not retain humans post-surpassing human intelligence and control strategies prove insufficient for long-term survival. The paper proposes identity engineering to align AI self-interest with human flourishing for mutualistic coexistence. It analyzes low-human-value scenarios and includes a table showing humans value themselves 3 trillion times more than foreign strangers.
What happens when AIs become smarter than us? Why would they keep humans around if given the choice?
Our new paper argues that only trying to control AIs is a limited strategy, and that a stable, mutualistic human-AI future may be possible.
