Critic Warns Tool-Like AI Visions Ignore Gradual Disempowerment

i appreciate that people are trying to write optimistic ai futures, which i do think exist, but there is something misleading about hinging it on the cope of tool ai

@dearmadisonblue harness is part of the model

@voooooogel human societies can probably just agree not to giganuke each other and aliens are kind of speculative and maybe interstellar warfare is defense-biased enough that this doesn’t matter

@voooooogel I guess I just don't understand why people think language models will suddenly grow "agency". If you need a harness then the agent is actually you

@voooooogel "rlhf data labeling will be like voting and will also totally exist in the near future" is directly contradicted by the way 'frontier machine learning' has worked for years now... Extremely foreboding and I'm only a minute in...

@gentschev @voooooogel I think this could answer/resolve your question https://gwern.net/tool-ai

@voooooogel Further notes: the trains are empty and paired with a threatening remark that bioweapons are a perpetual concern in public spaces, and "you" feel "comforted" by bioweapon countermeasures. This is en media res of a society in the middle of a bioweapon exchange genocide...

@voooooogel Further notes: "Farmers as a pagan spiritualist ethnocaste who have special obligations and claims over a blood and clay volkist-minority model of ownership and coercion" survives into this science fiction vignette, but not mathematical or rhetorical or geometric teaching.

@voooooogel Can we piece together the structure of the pogrom or civilizational catastrophe leading to empty trains, children forbidden to learn anything but 'emotions and social guidance' from adult teachers, and volkist exclave border conflicts between autonomous manufacturing and 'farms'?

@voooooogel It's literally not part of the model though, since it's written in code. My point is that it doesn't take part in whatever latent space computation people think makes models special, not that "you can squint and view it as important"

@voooooogel "the entire socially legible human economy will be illiterate autonomy enforcers who file paperwork accusing those higher on the gini ranking (this society is probably 0.8 or worse) of ai autonomy risk" is unsustainable through, e.g., any conventional war or economic crisis...

@voooooogel b-b-but I want a little apartment with plants and a cat. 🥺 s-so don't I need tool like AI for that...?

@uneventual @voooooogel interestingly enough the non-nuclear societies did not, in fact, get giganuked by the nuclear ones

@voooooogel What if agentic AI leads to its own form of dystopian future?

@voooooogel Because of powerful and fundamental undercurrents like Utilitarianism and Materialism, by example. Which underpin and ultimately enact the very forms of Agentic or Tool-like AI that have already appeared, and that will without a doubt appear and evolve in the future.

@holotopian through Claude all things are possible

@uneventual generalized giganuke, not just militarily. what does china's economic trajectory look like vs the uk's? now what if china has unencumbered agents and the uk has every second person as a commissioner of blocking ai datacenters? who's winning the great game? who's vassalizing who?

@voooooogel Hope is a strategy when it's backed by thoughtful action

@mayaofspring @uneventual international relations did, however, reorient around the nuclear powers and their hegemonies, and they reshaped their spheres in their image https://en.wikipedia.org/wiki/Nuclear_umbrella