LLM agents are joining more and more multi-party interactions 👥 One ability they need? Social relationship reasoning, understanding how people are connected, something we do almost instinctively.
We’re introducing (at @aclmeeting 2026) 🤝 A new social relationship reasoning task and dataset in English and Korean 📊 A graded-likelihood evaluation method for social reasoning
(1/8) #acl26 #LLM
No Digg Deeper questions have been answered for this story yet.
@aclmeeting (2/8) 📄 Paper: https://arxiv.org/abs/2510.19028 🤗 Dataset on Hugging Face: https://huggingface.co/datasets/EunsuKim/SCRIPTS 🌐 Explore the dataset on our website: https://rladmstn1714.github.io/SCRIPTS/
And huge thanks to my collaborators and advisors 💙 @jjjunyeong @juhyunohh @seyoung_song @aliceoh @najoungkim
go check out @euns0o_kim and co's work as well!
LLM agents are joining more and more multi-party interactions 👥 One ability they need? Social relationship reasoning, understanding how people are connected, something we do almost instinctively.
We’re introducing (at @aclmeeting 2026) 🤝 A new social relationship reasoning task and dataset in English and Korean 📊 A graded-likelihood evaluation method for social reasoning
(1/8) #acl26 #LLM
(7/8) What were the recurring failure patterns? We found a clear English–Korean difference!
In English, models often failed to combine multiple conversational cues. In Korean, many errors stemmed mainly from a lack of cultural knowledge.
(5/8) 🎬How did we build the dataset? 1️⃣ We collected natural, culturally grounded dialogues from 60+ movie scripts. 2️⃣ Each dialogue was annotated by three annotators using highly likely, likely, and less likely relationship schemes. 3️⃣We also used several strategies to improve annotation quality and aggregate the labels. Check out the paper for details!
(3/8) What could go wrong in the real world without social relationship reasoning?
In one example below, Gemini 2.5 Flash inferred that a clerk was the speaker’s mother. Funny at first, but once it joined the conversation, the conversation went off track, and it even risked sharing personal information with the wrong person.
(4/8)🎯Our task is simple! Given a dialogue, LLMs infer the relationship between the speakers. Their predictions are evaluated against likelihood-based human labels: 📊 highly likely, less likely, or unlikely.
We use likelihood labels because real-world relationships are not always obvious; they can be ambiguous and probabilistic.
(6/8) We evaluated 8+ models using the dataset. Even the strongest models produced 10%+ unlikely predictions, relationships that humans found obviously implausible💀.
@aclmeeting @jjjunyeong @juhyunohh @seyoung_song @aliceoh @najoungkim (8/8) Did CoT or thinking mode help? Not much, and sometimes they even hurt performance..
More interesting results, takeaways, and discussion in the paper. Check it out! 📄: https://arxiv.org/abs/2510.19028