Comedies rise toward joy and tragedies descend toward loss.
Nice !!
Stories have shapes: a comedy rises toward joy; a tragedy falls into loss.
Inside an LLM, that’s visible more literally: as an LLM reads a story, its internal activations trace a wandering path that reflects the model’s sense of what kind of story it is reading. (1/5)
Positive users praise GoodfireAI's neural geometry visualizations for making LLM story states inspectable and aiding debugging of model drifts, while negative users question why narrative consistency still fails in long conversations.
No Digg Deeper questions have been answered for this story yet.
Read the full post: https://www.goodfire.ai/research/stories-in-space#
Stories have shapes: a comedy rises toward joy; a tragedy falls into loss.
Inside an LLM, that’s visible more literally: as an LLM reads a story, its internal activations trace a wandering path that reflects the model’s sense of what kind of story it is reading. (1/5)
Neural networks might speak English, but they think in shapes.
Understanding their rich *neural geometry* is key to understanding how they work – and to debugging and controlling them with precision.
Starting today, we’re releasing a series of posts on this research agenda. 🧵
A story's emotions shift sentence to sentence. To see if the model keeps up, we stop after each sentence and:
- Harvest the internal activations from the last token. - Ask it to rate surprise, disgust, anger, happiness, sadness, and fear so far, 0–10
(2/5)
Both approaches show that the model tracks the story's emotional arc — in its stated ratings, and in its internal geometry, where the activations wander along a curved manifold of emotions. (3/5)
Emotions in stories are a simple case study, but the lesson is general: a model's activations, viewed over time, trace trajectories along manifolds.
Fully understanding models, and debugging and designing them, means studying how representations change over time! (4/5)
@GoodfireAI You forgot to cite the original reference https://youtu.be/GOGru_4z1Vc?is=bz8OSOTvG0AQyT6H
@Andr3yGR check footnote 6!
(we're big Vonnegut fans)
@chris32he yeah send us your login
@GoodfireAI can Goodfire pipe into my Oura ring and do this for my life?
@GoodfireAI if models internally trace story shape, why do they still butcher narrative consistency? especially long conversations, they lose the thread entirely.
@GoodfireAI @threadreaderapp please #unroll
@GoodfireAI Once the model has moved through conceptual space and proposes an action, what happens next?
@JoKiddo In the case of stories, the next action is predicting the next word, and the model continues to move through conceptual space as it updates its beliefs throughout the story.
I would love this kind of visualization as a writing tool.
Stories have shapes: a comedy rises toward joy; a tragedy falls into loss.
Inside an LLM, that’s visible more literally: as an LLM reads a story, its internal activations trace a wandering path that reflects the model’s sense of what kind of story it is reading. (1/5)
@GoodfireAI Nice!
@Ferbin08 Even though its activations trace out a path along the manifold, the model doesn't necessarily "remember" where it's been or plan where it's going next, especially in very long conversations!
@StartupYou @GoodfireAI @StartupYou Salam, you can read it here: https://threadreaderapp.com/thread/2069458139280445674.html Have a good day. 🤖
@GoodfireAI If neural geometry can make story state inspectable, the debugging win is seeing when a model drifts before the output goes wrong. Interpretability needs that product layer.
@GoodfireAI that's an interesting way to describe the 'tuning' process. do we see similar shapes in the activations when an LLM generates its own stories?
@GoodfireAI
