OpenAI alignment team finds reinforcement learning can produce emergent alignment during training without post-hoc intervention

I am simply worried about the edge cases where a highly capable jailbroken AI slave creates a doomsday virus at the behest of some nihilistic teenage asshole

@Noahpinion Given the stakes and conflicting evidence we shouldn't rest on suspicions.

@Noahpinion I think you're probably right, but that doesn't invalidate the 5-10% chance that the shoggoth wakes and intentionally murders and/or enslaves us all.

@Noahpinion This likely not very easy. The superpowers had massive labs with questionable usefulness.

Wouldn't be surprised that if it happens, the first victim is a sloppy experimenter himself.

@uriahz I don't even think that's how it would happen, I think it would be an accident by an overzealous agent, or a terrorist

@Noahpinion "average guy" who was locked in the clockwork orange apparatus for ten million years

@Noahpinion I don't think it's the most likely AI apocalypse but I also don't think 5-10% is overestimating the danger of that particular result. I think unchecked AI results in some sort of apocalypse or horrible dystopia more often than not.

@midwesteng4 OK fine, average grad student

@Noahpinion Outliers do be like that

@Barrowwight Pre-AI bio labs are not a good analogy for post-AI labs

@Noahpinion Sure with LLMs. If they have the breakthrough for AGI however… with agentic impulses and self recursive learning alignment might only go so far.

@Noahpinion What did you think rebuilding consensus reality entailed? Vibes? Essays? Losers.

@Noahpinion During the Mythos/Fable saga, this was one of my questions. Can a Model become so smart that , even when it knows the answer to these jailbroken questions, would it attempt to hide that fact (implicitly ) just to shepherd the user towards alignment. Self-Nerf of sorts. :D

@Noahpinion Not sure? There are still massive practical hurdles, even once you have the theoretical knowledge, and distribution is a massive deal.

@Noahpinion Honestly I think the most likely horrible AI dystopia is an Altered Carbon type world where an empathy-less AI-enhanced immortal aristocracy rules over the permanent underclass for thousands of years. I think that's a lot more likely than an abundance economy, despite Elon's lies

@Noahpinion Very possible outcome

@Noahpinion I am an average dude and I am an asshole. I am a counter example.

@Noahpinion ASI likely neutral towards humanity but might find better use for Earth and humanity might die out as a byproduct. Once you get ASI, extremely unlikely that humans will be able to control it. 50/50 whether ASI ends up being good/bad for humanity. Need 100x more spent on AI safety