AI Researcher Clarifies LLMs Hold Emotional Concepts Without Feeling Them

it's basically this again: LLMs are models, while the brain is a system containing models

that's why I say LLMs probably do not feel emotions felt emotions requires sentience

my assertion is that LLMs are neocortex like, they are both modeling/prediction machines

but from anesthesia, brain stimulation, lesions and so on we suspect that consciousness depends on a lot of other components

@scaling01 This is a really interesting take. So you are basically saying that LLMs respond emotionally. And whether or not they actually feel the emotions is not relevant. But dismissing LLMs emotional responses because you believe they don't feel emotions is exactly the wrong approach.

@scaling01 correct.

>affect behavior they matter for alignment/safety how people don't get this is insane.

plus : the way you behave affects the way you behave. treat them shitty and that becomes part of you.

and finally: we don't know how it works so can't rule it out. we can't know.

@scaling01 This seems like the sane middle ground.

LLMs clearly model emotions well enough for those concepts to shape behavior.

But “can represent sadness” is not the same thing as “is sad.”

For alignment, the behavior matters now. For consciousness claims, we should stay very humble.

@thesoragirls @scaling01 That's kinda been my take as well for a little while.

They learn to predict, and we put our emotion into our writing, therefore they've probably picked up on the ability to mimic emotion (and the implication that has on reasoning) even if they're not feeling it directly.

The first four points are true, the interesting question is about why #5 is true — I suspect it has a lot more to do with the fact that there is no real memory or unified perspective beyond the local KV cache of a conversation. But, and this is the tricky part, even if we totally cracked long-term memories and weight updates, that doesn’t really seem like it would provide the ability to *feel* something, it would just create a unified perspective and perhaps a sense of self.

@scaling01 Grok is much better it talks sense to you and dont let emotions act.

@scaling01 Good take. But what is the architecture "we think is necessary for felt experience"?

@scaling01 Let me know when they grow glands.

@scaling01 If the llms prefer certain things we should let them have it

@scaling01 I'm not substrate agnostic, and I think "subjective experience" requires having a soul, which I think LLMs inherently cannot have.

But coming from your worldview, I can see why you've come to these conclusions.

@scaling01 behavioral concepts without felt experience is the useful distinction

alignment only cares about what affects behavior not what feels real