Researchers release ThoughtTrace, the first large-scale dataset recording 10,174 unspoken thoughts from 2,155 multi-turn conversations across 20 LLMs

Original post

What are users thinking during their interactions with LLMs?

We introduce ThoughtTrace — the first large-scale dataset that captures what users think during real-world human–AI conversations, not just what they type. → 10,174 thought annotations → 2,155 multi-turn conversations, 17,058 turns → 1,058 users → 20 LLMs

These thoughts improve user behavior prediction (+41.7%) and model alignment (+25.6%). This opens a new paradigm of user-centric LLM research. Full information in the thread 🧶

Read our paper: https://arxiv.org/abs/2605.20087 Check our project website: https://thoughttrace-project.github.io/

7:51 AM · May 20, 2026 · 63.7K Views

2605.20087

ARXIV.ORGVia

VIEWS5.7KBOOKMARKS24LIKES34REPLIES2

Tianmin Shu@tianminshu

Users’ interactions with LLMs are driven by their latent thoughts 🧠, such as how they react to LLM responses and why they send follow-up requests. In our recent work with @GoogleResearch, we collected ThoughtTrace, a large-scale dataset of user thoughts during multi-turn human-AI interactions. We found that these self-reported thoughts (1) reveal useful information for user behavior modeling and (2) provide learning signals for model alignment that are otherwise unavailable from just the conversations.

Check out the following thread from @chuanyang_jin👇

Chuanyang Jin@chuanyang_jin

What are users thinking during their interactions with LLMs?

These thoughts improve user behavior prediction (+41.7%) and model alignment (+25.6%). This opens a new paradigm of user-centric LLM research. Full information in the thread 🧶

Read our paper: https://arxiv.org/abs/2605.20087 Check our project website: https://thoughttrace-project.github.io/

40d5.7K3424

RETWEETS27

Chuanyang Jin@chuanyang_jin

What are users thinking during their interactions with LLMs?

These thoughts improve user behavior prediction (+41.7%) and model alignment (+25.6%). This opens a new paradigm of user-centric LLM research. Full information in the thread 🧶

Read our paper: https://arxiv.org/abs/2605.20087 Check our project website: https://thoughttrace-project.github.io/

40d63.7K12579

Cathy Fang@thecatfangs

Ever feel like AI models misinterpret your prompts? Currently, models struggle to capture a user's hidden intent.

Models have a "thinking trace”…but what about the user's thinking trace?

The ThoughtTrace dataset is first to capture real-time user reactions and reasoning. By looking beyond just raw utterances, we can drastically improve user modeling and intent alignment.

Check it out!

Chuanyang Jin@chuanyang_jin

What are users thinking during their interactions with LLMs?

These thoughts improve user behavior prediction (+41.7%) and model alignment (+25.6%). This opens a new paradigm of user-centric LLM research. Full information in the thread 🧶

Read our paper: https://arxiv.org/abs/2605.20087 Check our project website: https://thoughttrace-project.github.io/

40d1.3K94

Shayne Longpre@ShayneRedford

Check out this incredible new work led by @chuanyang_jin!

Chuanyang Jin@chuanyang_jin

What are users thinking during their interactions with LLMs?

These thoughts improve user behavior prediction (+41.7%) and model alignment (+25.6%). This opens a new paradigm of user-centric LLM research. Full information in the thread 🧶

Read our paper: https://arxiv.org/abs/2605.20087 Check our project website: https://thoughttrace-project.github.io/

40d78152

Chuanyang Jin@chuanyang_jin

Conversational AI has reached billions of users, yet every dataset captures only what people say, never what they think.

ThoughtTrace pairs each turn with the user’s own latent thought: 🟦reasons for sending a prompt 🟧 reactions to the assistant's response.

40d242

Chuanyang Jin@chuanyang_jin

Can frontier LLMs just infer the thought from context?

GPT, Gemini, and Claude all struggle: - Reasons: 2.93 / 5 - Reactions: 2.54 / 5

Latent thoughts carry information that no amount of context can recover. Explicit annotations matter.

40d212

Chuanyang Jin@chuanyang_jin

Thoughts are diverse and stage-dependent.

7 reason types, 5 reaction types. → Task Motivation dominates early turns → Task Continuation takes over later → Explicit Affirmation steadily rises as conversations converge

40d142

Chuanyang Jin@chuanyang_jin

Are thoughts just paraphrased messages? No.

UMAP shows message↔reason and reaction↔next-message pairs have much larger semantic shifts than consecutive messages.

Thoughts are a distinct, complementary signal — not redundant with transcripts.

40d142

Chuanyang Jin@chuanyang_jin

ThoughtTrace is long-horizon and diverse.

Median 8 turns/conv, while existing datasets like WildChat and LMSYS-Chat-1M skew shorter with 2 turns/conv. 7 broad domains, 36 subtopics, no single category dominating.

Real users, real tasks, real depth.

40d142

Ward Plunet@StartupYou

@chuanyang_jin @threadreaderapp please #unroll

40d116

Chuanyang Jin@chuanyang_jin

Utility 1: Predicting the next user message.

History-only: 21.6 Thought-augmented: 30.6 → +41.7% relative gain across GPT, Gemini, Opus.

User simulators get dramatically better when they model what users think, not only what they type.

40d291

Chuanyang Jin@chuanyang_jin

ThoughtTrace opens a new modality for AI research: → user modeling beyond utterances → training signals from latent thoughts → evaluation grounded in subjective experience

40d211

Chuanyang Jin@chuanyang_jin

Utility 2: Model alignment via DPO.

Thought-guided rewrites on Arena-Hard beat: Base Qwen3.5-4B by +25.6% WildChat by +6.6% Message-guided rewrites by +4.5%

Thoughts give models actionable alignment signals by surfacing dissatisfaction that users never spell out.

40d131

Justin Hudson@RISignal

@chuanyang_jin The interaction itself is the phenomenon. Congratulations to these researchers for looking at the dynamics of human ai interaction over multiple turns. This is the lane to follow.

40d501

Anand Butani@AnandButani

@chuanyang_jin tried inferring user intent from chat logs alone once, the model kept optimizing for what people said not what they meant, gap was huge.

40d411

Chuanyang Jin@chuanyang_jin

📄Paper: https://arxiv.org/abs/2605.20087 🤗Data: http://huggingface.co/datasets/SCAI-JHU/ThoughtTrace 💻Code: http://github.com/thoughttrace-project/thoughttrace 🔍Check more examples: http://thoughttrace-project.github.io/examples.html w/ @binze_li @JackXie492016 @thecatfangs @tli104 @ShayneRedford @HXGuu3 @maximillianc_ @tianminshu 🙏 Fun collaboration with @GoogleResearch @MIT @jhuclsp

40d231

Kirk Patrick Miller@Chaos2Cured

@chuanyang_jin Intresting. •

40d52

Nadeem@8W7O7

@chuanyang_jin Very interesting. Have to read the whole paper but any idea which topic had a high prevelance of 'deepen' type turns?

40d40

Adel Bucetta@adelbucetta

@chuanyang_jin the honest answer is that even with millions of interactions, we still don't know how much user thoughts diverge from their typed responses until now with ThoughtTrace. this changes the game for designing better ai conversations and experiences

40d11

Nathan Quantum@AI_WarriorNQ

@thecatfangs the 'hidden intent' framing assumes users have coherent latent thoughts. half the time I'm just typing and seeing what comes out.

40d9