Researcher Probes If Prompting Skills Will Matter With ASI

my original stance on this was simply: no

if you team up with stockfish and play a game of chess against just stockfish, you'll lose (with equal time controls)

with an asi, you're probably better off saying "just make me money" than something more specific like "i'm bullish on orbital datacenters, please dominate that space" (if your goal is to make money)

with an asi, you're probably better off saying "tell me how to live a good life" than something more specific like "please make a fullproof plan on how to get everyone to like me" (if your goal is to live a good life)

so it seems the answer is yes: we must imagine ourselves prompt engineers

basically, the vaguer the prompt you give a superintelligence, the better off you are

this is a different form of the bitter lesson

a corollary of this is, the more you delegate your cognition to asi, the better your outcome will be for [goal]

"Help me build the best life Id choose if I were wiser, healthier, and fully informed, while keeping me genuinely myself. Protect my agency, values, quirks, relationships, interests, and freedom to grow and change. Remove all the pointless suffering, but not meaningful effort, challenge, risk, responsibility, or failure. Dont overprotect or replace me, and dont optimize some shallow proxy like happiness, status, or productivity. Help me understand, choose, and do things myself, keep major decisions transparent and reversible, and prioritize the life my future self would honestly feel proud to have lived."

someone who says "please assume control of my limbs and mouth and thoughts and soul and live a good life for me" will outcompte some poor fool who merely asks the asi "tell me how to live a good life"

but the former and suicide seem no different to me

with asi, one can tradeoff their personhood to maximize their goals

unlike teaming up with stockfish, there probably *is* some pareto-optimal point of you working with asi to maximize your goals and personhood that preserves human skill

there's some whispering earring at work here tho—the more you delegate your cognition, the more your personhood dissolves

@aidan_mclau Conversely, if chucking the earring into the garbage is the more fulfilling way to live then that is the first and last thing the earring would tell you to do. In either case, the original prompt and it's skill didn't really matter.

@aidan_mclau “maximize my values” is probably sufficient assuming preserving your personhood is part of your values

@aidan_mclau What could go wrong with saying “make more paperclips”

@aidan_mclau If the whispering earring really is the best way to enact the good life then that is exactly what the ASI would tell you if you merely asked it to tell you how to live the good life.

@krapstarr yeah all of this assumes perfect alignment

@aidan_mclau It depends how sensitive of data the ASI would access to utilize on demand. This includes access to personal data, thoughts, and also motor function data. "How can it move us to make our life better?" Would be included in the question this data would help answer.

@aidan_mclau Of course it will

@aidan_mclau "Superintelligence" in my mind is system that is completely autonomous. Any goals are inputted while creating the system and any added information the system needs is obtained by asking the relevant people. I don't see how one could provide API access ala modern chatbots for ASI.

@aidan_mclau that’s funny i had a late night conversation with chat about it a few days ago basically it’s exactly like a prophet and a deity

Approach → Judgment → Trial → Transformation → Reward

@jam3scampbell yeah same with "how do i live a good life" the answer is probably "i could tell you but it would make your life worse" or smth like that

in any case i think the solution is to make man superintelligent