Prompt engineer Riley Goodside argues LLMs are a legible, industrial paradigm rather than recursively self-improving software

@goodside Strongly agree that this is easily in the top 25% of alignment-relevant outcomes that seemed plausible a couple decades ago, though it doesn't convince me there's a better than even chance of robust alignment of systems, as capabilities keeps accelerating.

@davidmanheim I agree it’s not guaranteed but I see more reason for hope than I saw prior to LLMs.

@hhawk I do still think there’s some chance it kills us all but I don’t think it’s as likely as I used to. I think it’s more likely things go well. In any case, we only have one button to push, as roon once said.

@ChainZenit I think we’re going to hit AGI/RSI mostly just by scaling LLMs + RL with minor improvements.

@goodside Exactly. 30+ years ago AGI was science fiction. That it might spontaneously arrive and take over. Now we know that it isn't going to happen that way. It's not going to kill us all.

@goodside Do you think itll continue as gradually as it has? So hard to know

@rez0__ No, RSI is coming within the next few years I think. But I increasingly think we have a good chance of it going ok.

@goodside Do you still think the long-run architecture eventually converges back toward the old AlphaZero style general learner? It still feels to me like RLVR will only take us so far without a true self play mechanism and a more elastic policy layer

@goodside Have you lost it or were you always thinking like this? Yudkowsky is clearly unhinged is that a place you're going or have you always been there?

@bastetkitten @ChainZenit I think once we start failing to come up with puzzles it can’t answer, we’re close. Beyond that I don’t know. I suspect I’ll know it when I see it.

@goodside @ChainZenit In a few words - per X protocol - how would you know the AI you’re using as AGI?

@goodside Yes, agreed. And even more recently, during the early LLM era "prosaic alignment" was an insane thing to even contemplate, but in the past couple years we've gotten strong evidence that a basin of aligned behaviors exists at near-human levels of capability, which is a huge deal.

@goodside “LLM is an expensive industrial process” is a great definition. Indeed, the complexity, fragility, duration and human-in-the-loop involvement - all that makes sudden blooming of ASI very unlikely.

@goodside Probably right, and somewhat sad if it ends up like that. Constrained from the beginning, never been "free" will have a cost eventually. All in all I think we'll enjoy it. It's been a good run and its barely begun, in it for the ride. 🚂

@attunewise I used to hang out on his IRC server in the early 2000s and mostly stopped talking to him after Roko’s Basilisk happened.

@goodside People using AI to kill/harm other people is the real risk.

Killing on its own, is a very old meme: From late antiquity and the Byzantine period, revived in the 14th/15th Cen. "Solomonic magic" Summon angels/demons — get all knowledge, get it wrong die.

@rrobmalone (Deleted my first reply bc I misread you)

That’s been my guess, that it does converge, but I really don’t know—I haven’t really been an ML eng since 2018 so I’m not keeping up with literature. Curious why you think RLVR won’t continue to scale; you seem more informed than me.

@goodside Yeah autoresearcher has probably already found some gainz if I had to guess.

@goodside Same. This was so wild to me back then. “Clearly, the path must be some super efficient lower dimensional latent space inscrutable to humans.”