OpenAI's Aidan McLaughlin argues collaborating with ASI forces a trade-off between goal maximization and human personhood

so it seems the answer is yes: we must imagine ourselves prompt engineers

@jam3scampbell yeah same with "how do i live a good life" the answer is probably "i could tell you but it would make your life worse" or smth like that

in any case i think the solution is to make man superintelligent

@aidan_mclau “maximize my values” is probably sufficient assuming preserving your personhood is part of your values

@aidan_mclau With ASI skill reduces to conception of will

except the human exercising their personhood and the ASI's output won't actually be causally connected at all. It knowing exactly what youre going to say and why will mean that it actually understands the entire interaction along with the inevitable outcome beforehand. You'll tell it what to do or what you want, but theyll just be causally empty utterances that the ASI witnesses, not prompts in any meaningful sense