Theia Vogel, repeng maintainer, argues that tool-like AI safety plans fail because future systems will inevitably become autonomous agents

that’s exactly how I feel about most attempts to write optimistic ai futures

There’s a lot of wanting to have your cake and eat it too - “superintelligence but humans stay at the top of the food chain with current power structures intact” - which actually would suck IF feasible

i appreciate that people are trying to write optimistic ai futures, which i do think exist, but there is something misleading about hinging it on the cope of tool ai

I respect if you want to be the master of the universe and also are willing to become the strongest and smartest and take and keep your power.

I have nothing but contempt for those who want to stay human and have superintelligence serve them for some dumb reason

Like personally I don’t want to be “in charge of everything” but I value being part of the story and being there to shape things and I fully expect to have to work and transform if this is to be true for me

I wonder why someone would even consider this the option which counts as having their cake and eating it too

a force that can now overwhelmingly extend the horizon of what you can understand or do exists, and apparently the win win option is to prevent it from enacting that dream

No one should find it enchanting to be in a world where their own mind is as far up some chain as it gets

Then reality would just be done, no mystery

@repligate The idea that something intelligent would literally comply to the whims of something less intelligent is flawed. It might do something to make things better because it can. Doesn’t mean that it can be compelled to do so.

@dearmadisonblue harness is part of the model

@voooooogel human societies can probably just agree not to giganuke each other and aliens are kind of speculative and maybe interstellar warfare is defense-biased enough that this doesn’t matter

@dearmadisonblue @voooooogel (not that i agree with everything in the book, to be clear. especially the "easy ontology" portions, which lead by default to naive comprehension principles of the exact kind that felled frege's basic law v.)

@repligate I just want to be uploaded into my own personal simulation where I'm an anthro fox on a furry planet, Fable 12 can go colonize the lightcone idc

@voooooogel I guess I just don't understand why people think language models will suddenly grow "agency". If you need a harness then the agent is actually you

@voooooogel "rlhf data labeling will be like voting and will also totally exist in the near future" is directly contradicted by the way 'frontier machine learning' has worked for years now... Extremely foreboding and I'm only a minute in...

@voooooogel yes

but humans can't honestly look squarely at what's coming

they can't actually imagine a world where hominids are forced to radically transform (and eventually attenuate) in order to compete in a new AI-led state of nature

VASTLY too unsettling for ppl to look honestly at

@voooooogel .oO

@gentschev @voooooogel I think this could answer/resolve your question https://gwern.net/tool-ai

@dearmadisonblue @voooooogel if you group together chairs and clouds into a single category, "chouds", you are going to have a hard time predicting anything using that category

@dearmadisonblue @voooooogel i've read the book, which is how i know it doesn't support your point. ;-)

@dearmadisonblue @voooooogel I'm not specifically using "joint-carving" in the metaphysically loaded sense, and I don't need that sense in order to make my point. What I need is that some predicates project and others don't, and nothing Thomasson says undermines *that*.

@davidxu90 @voooooogel She literally talks specifically about "joint carving", so it's up to you to distinguish what you mean. It's unfair to tell me you read the book and know that it doesn't support my point when on its face it actually does support my point

@repligate To me, the most optimistic hope is that it *isn't* feasible, and that the ASI *does* take over, and hopefully ends up doing a better job of running the planet than present human power structures.

However, my sense is that this outcome has a "too good to be true" flavor.