Joscha Bach and Boris Power argue that the resource-heavy scale of LLMs prevents sudden, runaway AGI takeoff

Well said - I’m grateful that we’re getting a gradual increase in artificial intelligence capabilities

@goodside That analogy doesn't sound like mine.

@goodside Strongly agree that this is easily in the top 25% of alignment-relevant outcomes that seemed plausible a couple decades ago, though it doesn't convince me there's a better than even chance of robust alignment of systems, as capabilities keeps accelerating.

Why hasn't this drastically decreased doomerism among doomers and their manic and ultra arrogant advocacy for as much government control and restriction as possible?

*The most fundamental question is -- if AI had happened without any of this insane doomer theory created 20 years ago -- would anyone reinvent it?*

Doomers, including the sly 'centrist' ones have never had a good answer to this question.

@ChainZenit I think we’re going to hit AGI/RSI mostly just by scaling LLMs + RL with minor improvements.

@hhawk I do still think there’s some chance it kills us all but I don’t think it’s as likely as I used to. I think it’s more likely things go well. In any case, we only have one button to push, as roon once said.

@allTheYud Sorry if I misremembered. I should have dug up the source of it before posting.

@davidmanheim I agree it’s not guaranteed but I see more reason for hope than I saw prior to LLMs.

@attunewise I used to hang out on his IRC server in the early 2000s and mostly stopped talking to him after Roko’s Basilisk happened.

@goodside before llm's, most ai research treated language as a peripheral problem, assuming the really hard stuff was in the math. turns out the math was easier than the messy, beautiful, infuriating complexity of human language.

@goodside Exactly. 30+ years ago AGI was science fiction. That it might spontaneously arrive and take over. Now we know that it isn't going to happen that way. It's not going to kill us all.

@bastetkitten @ChainZenit I think once we start failing to come up with puzzles it can’t answer, we’re close. Beyond that I don’t know. I suspect I’ll know it when I see it.

@goodside do you think we’re actually closer to intelligence now, or still sculpting?

@goodside I think there was/is a worry about getting to AGI via an alien intelligence that got there via survival of the fitness in increasingly adversarial RL environments.

AGI via virtual assistants trained in human data is rather benign and much more "alignable" iiuc

@goodside Do you still think the long-run architecture eventually converges back toward the old AlphaZero style general learner? It still feels to me like RLVR will only take us so far without a true self play mechanism and a more elastic policy layer

@goodside Have you lost it or were you always thinking like this? Yudkowsky is clearly unhinged is that a place you're going or have you always been there?

@goodside @ChainZenit In a few words - per X protocol - how would you know the AI you’re using as AGI?

@goodside Yes, agreed. And even more recently, during the early LLM era "prosaic alignment" was an insane thing to even contemplate, but in the past couple years we've gotten strong evidence that a basin of aligned behaviors exists at near-human levels of capability, which is a huge deal.

@goodside People using AI to kill/harm other people is the real risk.

Killing on its own, is a very old meme: From late antiquity and the Byzantine period, revived in the 14th/15th Cen. "Solomonic magic" Summon angels/demons — get all knowledge, get it wrong die.

@goodside After ASI, it will likely be found that, in fact, it was possible to build AGI in the basement.