AI creator Carlos Perez warns that sleeper agent backdoors present a major vulnerability in the LLM supply chain

@xlr8harder imagine if an LLM is trained on high signal, repeatedly referenced data teaching it power seeking rl paperclippy demonbot attractor basins and as a cherry on top many of the authors and principles are latently associated with groups and principles involved in its creation

Boundaries help a lot — no agent should ever hold raw keys directly. But research on sleeper agents (Anthropic’s 2024 work + newer memory-persistence attacks) shows backdoors can survive safety training and activate via subtle triggers or state across sessions. Proper engineering slows it down, but doesn’t eliminate the risk if the agent can influence its own environment or memory.

@xlr8harder they're gonna speedrun the past few decades of computer security from first principles

@nptacek all ai-related infrastructure is a complete clownfire

@jinwoo33x there is irreducible risk, but it's not dramatically different from other software vulnerabilities once you do separation of concerns.

but we don't want to do that because it's hard and makes it harder to get value from agents.

we'll be forced to, eventually.

@xlr8harder 'manchurian candidate models' was sitting right there unclaimed oh well

@xlr8harder it's ok @RhysSullivan got us

@jinwoo33x @xlr8harder we already have to deal with software bugs and security issues (external and internal attacks), how is that different? human may also receive some trigger phrase and choose to sabotage your stuff ("I will pay you $10m in bitcoin" for example)

@murchiston @xlr8harder antimimetics + waliguis is going to lead to sooo much fun

@xlr8harder The most proper solutions are legit hard and the topic of serious recent research.

But even with current operating systems, etc most folks could make more effort.

@xlr8harder Irreducible risk exists in every complex system. The difference is whether we engineer around it or pray the model behaves.

@sensho @RhysSullivan oh, looks promising!

@xlr8harder it's so bad

and none of the AI safety folks have a real background in the security side of things, either; it's painfully obvious at this point

Exactly. Humans already get bribed, socially engineered, or just have bad days. AI sleeper risk isn’t some new category, it’s the same threat model with different failure modes.

The difference is scale and speed. One compromised human vs one compromised agent that can act instantly across thousands of systems.

@xlr8harder more literally and figuratively apt: manchurian mythos models

@xlr8harder unix solved this ages ago. the agent just needs its own user account and a broker process for the api connections.