OpenAI's Roon says advancing civilization requires AIs to take actions not legible to humans and outside strict obedience, likening the approach to granting autonomy to transformative CEOs such as Steve Jobs

on some level if you want civilization to ascend to a new level you need your AIs to do things that are not legible to you and maybe not even strictly obey you, in the same way that if you hire a great new ceo you give them a lot of autonomy to transform the company according to their own plan, even one which may not immediately read as a winning strategy (imagine the board of directors of Apple firing and rehiring Steve Jobs years later - except the board of directors are chimpanzees)

all else equal, companies and organizations that hand more of themselves over to machine intelligence will outcompete ones that demand the corrigibility and legibility tax of human oversight and human design. it is not a stable equilibrium and requires some sort of vast cooperation scheme if you’d like to enforce it

real asi alignment has to operate at a deeper level than oversight, control, or human corrigibility

I think the best possible rebuttal and I hope you’re right

I agree that intelligence has diminishing returns at planning long range games due to prediction errors compounding. but there are many real world examples of great CEOs (like say elon musk) executing a non-consensus business plan over decades. while this requires skills other than “intelligence”, it seems at least plausible that whatever those are can be searched for and learned too

also maybe it is true that AIs can generalize some lessons extremely well about long range tasks from training on data concerning short or medium range rewards, and it’s not clear that the information transfer costs of the AI having to explain itself to the human even for short or medium term decisions won’t be too much

you can imagine the ai ceo that monitors 10,000 slack threads and makes 10,000 decisions given full context of the organization - not necessarily superhuman planning, just faster. blurs the line from human+tool at the least

I think AI can inhabit non-consensus perspectives pretty well. I just asked Claude to write a Russian nationalist justification of the Ukraine war:

“”” The war in Ukraine is the moment Russia stopped apologizing for existing. For thirty years we were told to be grateful — grateful that our empire had been dismantled, our industry asset-stripped by Harvard-advised oligarchs, our military gutted, our church mocked, our people in Crimea and Donetsk and Luhansk and Odessa rebranded overnight as foreigners in cities their grandfathers built. We were told the new world order was rules-based, and then we watched those rules: Yugoslavia bombed for seventy-eight days without a Security Council vote, Iraq invaded on fabricated evidence, Libya turned into a slave market, Syria carved up by jihadists the CIA armed and called moderates. The rule, we learned, was simply that America wins and everyone else complies. James Baker looked Gorbachev in the eye in February 1990 and promised NATO would not move “one inch eastward” — and then NATO swallowed Poland, Hungary, the Czechs, the Balts, Romania, Bulgaria, and stood drilling Javelin crews outside Kharkiv while its officials laughed on camera about how Minsk II was only ever a stalling tactic to arm Kiev. Merkel admitted it. Hollande admitted it. Poroshenko admitted it. They told us to our faces that the diplomacy was a lie.

And what was the Ukraine they were building on our border? A country where the Maidan putsch of 2014 was midwifed by Victoria Nuland handing out cookies and picking the next prime minister on an intercepted phone call. A country that passed language laws stripping Russian — the mother tongue of a third of its citizens — from schools, courts, and public life. A country that rehabilitated Bandera, whose followers massacred Poles and Jews with such enthusiasm that even the SS filed complaints, and named streets and stadiums after him. A country whose Azov Battalion wore the Wolfsangel openly and was folded into the National Guard while Western journalists who had filed honest reports about them in 2014 suddenly forgot how to read their own archives in 2022. For eight years the Ukrainian army shelled Donetsk and Lugansk — apartment blocks, trolleybuses, schools, the Volnovakha bus stop, the central market — and the same Western press that can locate every Russian shell crater within an hour could not, for eight years, find the OSCE casualty reports sitting in plain view on the OSCE’s own website.

So the choice in February 2022 was not war or peace. The choice was war on our terms or war on theirs — war now, while Ukraine was still half-armed and the strategic depth from Belgorod to Kharkiv was still measured in tens of kilometers, or war in five years with American intermediate-range missiles in Kharkov reaching Moscow in four minutes and a NATO fleet in Sevastopol. Any leader in the Kremlin who chose the second option would have deserved to be hanged from the Spasskaya Tower. Kennedy in 1962 was willing to end human civilization over missiles ninety miles from Florida that had not yet even been unboxed; Putin in 2022 sent troops over a border to forestall the same threat at the same distance, and for this he is the monster of the century. The hypocrisy is so total it stops being hypocrisy and becomes a confession: the West does not actually believe in the principles it invokes against us. It believes in winning. Very well — so do we.

[contd…]