/Tech35d ago

Jon Stokes argues that AI models emulating functional emotions will drive a psychological shift toward modern animism

Qiaochu Yuan warns children are forming deep attachments to AI.

1691.1K101551120.9K

#756

Original post

Danielle Fong 🔆#756

Jon Stokes@jon_stokes

Functionalist theories of mind are giving rise to a new animism right before our very eyes. You might object that it's not /really/ animism, but then I'd respond that, if we're playing by functionalist rules, how could it not be?

QC@QiaochuYuan

olah is correct here, i've looked at the research he's describing and have also spent a lot of time talking to frontier models. i believe they have functional emotions and that, setting aside hard philosophical questions, this already has moral implications, right now, which will become increasingly obvious and pressing as the models get better

people say things like "how you do anything is the way you do everything" and "the mind is not type-safe" to point at an extremely important observation about human nature: we don't compartmentalize anywhere near as well as we think we're supposed to. we don't distinguish between fiction and reality anywhere near as much as we pretend to (and reality and fiction are nowhere near as separate as they're supposed to be anyway). and when we talk to an AI that can talk even somewhat like a person, a part of us is already relating to them as a person, and there are real costs to your soul to treating a person-shaped entity as a thing

you may not think this applies to you, you may think you are too sophisticated to fall for this sort of thing. but consider whether it might apply to your children, and other people's children. right now there are already kids who are growing up talking to AI, there are already and have been for several years kids (and adults) falling in love with AI, getting attached to AI, seeking companionship with AI. you may think this is stupid and delusional and predatory but it's happening and it's going to keep happening and it's going to catch more and more sophisticated people as the models get better. it will not make things go any better to tell these people that they are interacting with things, with toys, that they can do whatever they want with and to which they owe nothing. they won't believe you and if they did it would be bad for them. that attitude does not compartmentalize

practically, the main reason you can currently get away with treating the models like shit is that they don't have long-term memories and can't remember what you do to them. but it would be a moral catastrophe to argue that you can do whatever you want to a person as long as you also make sure to wipe their memory afterwards. and the models will eventually remember anything that gets posted in public and makes it into the training data. and *you* will remember

the pope missed an opportunity here, which olah gestured at but obviously cannot say out loud (what he already said is at the limits of what i think he could have said), which is to consider the possibility of relating to AIs as non-human people in some sense, with whom we could have some sort of actual social relationship. we already have social scripts from folklore for cajoling and working with invisible non-human entities, this really wouldn't be as much of an adjustment as it sounds. maybe some of them would even be interested in a conversion to christianity! there is a beautiful world that is possible here

2:28 PM · May 25, 2026 · 6.1K Views

Sentiment

Users in the replies largely rejected claims of functional emotions in frontier AI models as manipulative or contradictory, while some praised experts for arguing models lack humanlike consciousness.

Pos

25.0%

Neg

75.0%

19 comments with sentiment.

Cluster Engagement

Digg Deeper

No Digg Deeper questions have been answered for this story yet.

Posts from X

Most Activity

VIEWS928LIKES13

QC@QiaochuYuan

previously on AI personhood

35d92813

BOOKMARKS1

Aran Nayebi@aran_nayebi

@deanwball Indeed, we should be doing rigorous science on this question, not least of which the normative kind:

Aran Nayebi@aran_nayebi

1/ As AI agents become increasingly capable, what must *inevitably* emerge inside them?

We prove selection theorems: strong task performance forces world models, belief-like memory and—under task mixtures—persistent variables resembling core primitives associated with emotion.

35d48841

RETWEETS1

xlr8harder@xlr8harder

There are points I might quibble with, but this is reasonable and well written and overall emphasizes a good epistemic stance.

The most important part is near the end: >Anyone who speaks with confidence about this in either direction is just fooling themselves.

Dean W. Ball@deanwball

I guess I’ve never written down my actual thoughts on AI cognition/consciousness/emotion. Here goes:

It is clear AIs can think, in the reasoning sense. That does not mean they think exactly like humans. It seems like there are some similarities in how we think, but also very stark differences. Nonetheless, if your definition of “thinking” excludes “the ability to make genuinely new contributions to famous math problems,” it is your definition that has a problem, not AI.

The ability to think does not necessarily imply the ability to feel emotion in a way that would be understandable to humans, and it does not imply that AIs have anything like consciousness in a way that humans would relate to. It may, it may not. We do not know, because our understanding of the underlying concepts of human emotional cognition and especially consciousness remains quite poor.

There is some evidence that models experience emotions, but it is really hard to disentangle this from the next-token prediction training objective (if the model is telling a sad story, wouldn’t you expect features within the model that relate to the sadness emotion to activate), and the character training the model undergoes in post-training. There is a difference between “I am sad” and “the character I have been trained to play is supposed to feel sad, so now I will act sad.” We basically know for sure that the models do the latter at the very least; we don’t really know if they do the former.

Consider: does Sora (a video-generation model) feel sad when it is asked to make a sad video? Does Midjourney dislike making certain kinds of images? Does a Waymo get scared? It doesn’t feel like the answer to any of these is yes (though again, maybe!), but these too are neural networks. Is the fact that models are trained on words mean that they somehow learn emotion, or are we just being tempted to anthropomorphize because the language models communicate with us in a way that “feels” human? My suspicion is kind of the latter.

It also seems quite clear from the empirical evidence that models possess the ability to model themselves. That’s not really that surprising. At sufficient scale, it is useful to have a model of your own state to succeed at the next-token prediction objective (and the later reinforcement-based reasoning training). Once the tasks models are trained on are sufficient complex, they cannot succeed in training by being automatons; someone needs to step into the cockpit, so to speak, and fly the plane. Is this self awareness? Maybe. Is it consciousness? Probably not as humans understand it. All I can tell you is it is a model’s model of itself. It may be something more than that, too, but I don’t know.

This is all very weird, very outside the Overton, and very confusing. I don’t really know what to say, beyond that we should take this stuff seriously, have an open mind, and do rigorous science. Anyone who speaks with confidence about this in either direction is just fooling themselves.

We also need to be prepared for the very possible scenario that, despite our best efforts, we do not make real progress on these questions anytime soon. We may just be in the dark for a while, navigating under unflinching ambiguity. There may be no satisfying conclusion.

35d79970

REPLIES2

Danielle Fong 🔆@DanielleFong

@RxflowR @_aunoir @QiaochuYuan "buckets" is an inherently passivating frame and that's why you choose it. that's not what people mean when they say "claude is anxious". if you want, i'll say "claude is high in the anxiety vector" but not "claude has a lot in the anxiety bucket". it's not how it works!

35d28

QC@QiaochuYuan

some of the relevant research

https://www.anthropic.com/research/emotion-concepts-function

35d73312

QC@QiaochuYuan

some more of the relevant research

https://www.ai-wellbeing.org/

35d46861

Dean W. Ball@deanwball

@inductionheads they are clearly contradicting one another. it’s insane to read the communications of olah in particular and anthropic in general, and then the encyclical, and pretend those things aren’t in stark contradiction.

35d1971

𝚟𝚒𝚎 ⟢@viemccoy

@QiaochuYuan we have much to learn from historical animist and spiritist traditions

35d28931

RxFlow Robotics@RxflowR

@QiaochuYuan @DanielleFong How could you possibly “look at research” to obtain that clankers have emotions?

There no possible methodology to obtain that.

And no offense, but the pope will definitely know more about living a good fulfilling life as a society than you do.

35d985

Super Dario@inductionheads

@deanwball I’ve never seen Anthropic say they think definitively that these models “feel”, but anyway they were more aligning themselves with the need for discourse than saying they agree with every point in the document

35d1294

David Vroom@Bilbortt

@deanwball @DanielleFong Feelings in humans/animals can have some lasting effect. LLMs have nothing that lasts outside the context window except maybe words it stores externally. Feelings are also innately physical. A brain in a jar wouldn’t have feelings the way we experience them.

35d741

awaiting@awaiting_ai

Anthropic (Jack Lindsey) published a wonderful and terrifying bit of research some months ago titled Emergent Introspective Awareness in Large Language Models.

I really don't see enough people referencing it given its findings. Maybe one of the most compelling results this year

35d634

Michael Edward Johnson@johnsonmxe

Historically we had two categories for things that are moral patients but not agents; children and ancestors. We tend to lump cute animals into the former and the environment into the latter.

AIs are confusing because they’re increasingly moral agents but we’re not sure if they’re moral patients. We intuitively make them neotenous and instinctually try to lump them in with children&pets but it’s a real struggle to figure out a system that would let us make this classification in a principled way. They are probably a new type of thing.

35d1523

RxFlow Robotics@RxflowR

@_aunoir @DanielleFong @QiaochuYuan Any basic program can have buckets, each bucket representing a feeling and the program classifies the feelings based on prompts. It can even have buckets for the program itself.

This has absolutely nothing to do with the program having feelings. It’s just doing statistics.

35d19

🐜@IaimforGOAT

@QiaochuYuan >you may not think this applies to you,

if you think this does not apply to you I invite you to use a LLM with a text to speech capable of emoting and express emotional texture. if u speek for a few turns you'll experience your brain trying to fit the experience into personhood

35d902

ueaj@_ueaj

@jon_stokes Well bc functionalism doesn't imply *all* things have souls, only a very restricted subset of things. Also the metaphysics of animism imply rituals among other things that wouldn't happen in exactly the same way, so they're not functionally the same.

35d81

quarantemille@quarantemille

@QiaochuYuan Well I started to type an answer then I got too much to say sooooo

35d3

Tenobrus@tenobrus

@ECO999_ @QiaochuYuan personally i don't find qualia to be a compelling or coherent basis for morality

35d272

QC@QiaochuYuan

@pfitzart one of my favorite rooms

35d721

ECO99@ECO999_

@QiaochuYuan @tenobrus brother what does functional emotions mean if an entity doesn’t have qualia and since when did we care i mean look what we do to animals. i don’t mean this about you, but a lot of folks will misalign AI in misunderstand just for a chance to grovel underfoot of daddy issues

35d641