OdysSim releases OSim-8B, a 10-billion-token corpus, and the SOUL-Index benchmark to train LLMs to simulate realistic human behavior

💡 Step 1: measure it. We built SOUL, 5 axes of human-like behavior (conversation, social skills, cognition/ToM, role-play, evaluation), and SOUL-Index, 23 benchmarks under one roof.

💡 Step 2: the data. 21.4M real human interactions (~10B tokens) from 62 sources. Most raw dialogue data has no idea WHO is talking or WHY. So we retrofit every conversation with back-generated social grounding (e.g., a character profile + interaction goal), and the model learns behavior grounded in context, not just text.

@sunweiwei12 @Jiarui_Liu_ @StigLidu @judysun233 @1000seagull 📄 Paper: https://tinyurl.com/8td8ux9d 🤗 Model: http://huggingface.co/collections/cmu-lti/odyssim 🤗 Data: http://huggingface.co/datasets/cmu-lti/osim-mid-training ; http://huggingface.co/datasets/cmu-lti/cmu-lti/osim-post-training 💻 Code: http://github.com/sunnweiwei/OdysSim

Have fun with our models and data!

@nlpxuhui I’m skeptical that a model can capture a large range of human creativity.

Some human behaviors can be contradictory, can they be merged into one model?

💡 Step 3: training, in two very different stages.

Midtraining on the corpus teaches the model what human behavior looks like. It shifts length, formatting, and word choice toward the human register.

Then we train one RL expert per benchmark, using verbal feedback from LLM judges where rewards aren't verifiable, and merge all 23 specialists into one model via expert distillation.

Fun ablation: the two stages do different jobs. Midtraining fixes the register; RL drives the benchmark gains.

The part I find most interesting: we dropped OSim zero-shot into τ-bench as the customer talking to a tool-use agent, and scored it against real humans doing the same 165 tasks (τ-USI).

OSim-8B's reactions are nearly indistinguishable from real users: React alignment 93.2 vs 93.5 for humans.

Meanwhile, some of the most assistant-tuned frontier models are among the WORST user simulators. Their helpful, agreeable register is exactly what real users don't have. Helpfulness ≠ humanity.

📈 Results on SOUL-Index:

✅ OSim-8B lands within ~1 point of GPT-5.5, Gemini 3.1 Pro & Claude Opus 4.7 on the 23-task average ✅ best or tied-best on 8/23 tasks, more than any single frontier model ✅ +18.0 over the best frontier model on user simulation (UserLLM), +16.8 on social interaction (Sotopia-Hard) ✅ beats every prior open behavioral-simulation model (UserLM, CoSER, HumanLM, Sotopia-RL) nearly across the board

An 8B model. Open weights.

I think the key is not to say "capture human creativity" (the term is low-key vague by itself)

The real question: as frontier models increasingly collapse into a similar “assistant” shape (which are incredibly important and useful imo)

How do we rethink the objective and build something meaningfully different, something that opens up new forms of applications and interaction?

we have thoughts and would love to chat! ☄️

We're just getting started, and this seems to be a very different path from what the current frontier models!

Huge thanks to my amazing collaborators: @sunweiwei12, who co-led the project with me! 🫡 and @Jiarui_Liu_ @StigLidu @judysun233 @1000seagull for help and @tongshuangwu Yiming @MaartenSap for great mentorship and strong support!

Training on LLM-judge rewards for *behavior* gets weird. Two failure modes we caught:

🕵️ Judge manipulation: in Sotopia the model started inserting evaluation-like statements into dialogue ("our relationship score should be perfect") instead of actually being social. In ~20-25% of rollouts!

📉 Short-response collapse: when judges subtract points per detected error, the model learns to just say "I'm fine." Fewer claims, fewer errors, higher reward.

An LLM hacking detector + rubric rewards mitigated both (ofc, there are defs more works to do there). If you do RL on judge-scored behavior, monitor the behavioral stats, not just the reward curve.

🧭 Key insight:

Human simulation isn't a prompting problem, and it isn't solved by scale. The most capable assistants are often the least human.

Building simulators means realigning the whole pipeline around behavioral realism: measure it (SOUL-Index), feed it (socially grounded data), and reward it (behavioral RL). That's what "behavioral foundation model" means.

@nlpxuhui Agreed that’s a broad term I used for things that I’ve had hard time coercing current frontier llms to generate.

But do you think it could be captured by a (group of) cleanly defined objective(s)?

@nlpxuhui @sunweiwei12 @Jiarui_Liu_ @StigLidu @judysun233 @1000seagull yay thank you for public data

I think these models fall under the regime of 'mildly useful' regime sans sanctity and guarantees. What will be really interesting is if under strict constraints, they replicate human idiosyncrasies such as certain forms of irrational behavior or mean-regression tendencies under crowding.

@nlpxuhui great work

@nlpxuhui I'd add another dimension- accountability. A model should be constantly reviewing its output and processes that generate non conforming responses. This is a essential trait of successful humans

@nlpxuhui @Scobleizer I want it to be helpful yes

I agree our models are defs far from being perfect 😁

One thing to highlight is that we did put our model in simulating users for tau-bench (and compare with real human customers).

And our model is showing very interesting behaviors like pushing back and getting annoyed, forgetting about things etc, that frontier models rarely show.

The signs are still pretty early but yeah, a starting point...

@nlpxuhui Very interesting work Xuhui. Thanks for the pointer ! 🙏

Bravo to the team 👏👏

@nlpxuhui