Reactions from ranked influencers
2 posts
Factuality and human preference are complementary signals. 00:00 Why we're adding factuality as a signal 00:14 Human preference vs. factuality: complementary, not the same 01:06 How this differs from style control 02:32 The math: reviewing the existing Bradley-Terry objective 04:27 Extending the objective to factuality 06:23 The composite objective: weighting human preference + factuality 07:14 Why rankings shift between human preference and factuality 07:34 Defining the factuality label 09:48 Why models aren't penalized for abstaining from claims 13:44 Confidence intervals via M-estimation and asymptotic normality 17:27 Wrap-up: what's next for factuality at Arena
Introducing factuality in the Arena: a new ranking of models according to a weighted combination of human preference and factuality. Model rankings are now viewable according to a weighted combination of human preference and factuality. Factuality is live in our Text and Search Arenas as a non-default toggle. We audit model responses by randomly sampling battles and extracting web-verifiable claims. We then verify these claims and compare the average correctness between model responses. To power these rankings, we’ve labeled over 2 million claims made by LLMs in real-world conversations, 1.3+ million from Text Arena, and 700k+ from Search Arena. Notable highlights with factuality enabled in the Text Arena: - Claude Fable 5 moves down slightly to spot #2 - GPT-5.5 saw the largest increase, moving up 13 spots into the #7 spot - Muse Spark dropped the most from #7 to #20 (-13pt) By labs, Meta saw the largest drop from #2 to #5, while Anthropic overall held the #1 spot. Looking at only open model providers, Xiaomi saw the largest improvement, jumping from #9 to #6. Learn more about the findings and methodology in this thread.
Technical deep dive into how we measure factuality in Arena. Check it out!
Factuality and human preference are complementary signals. 00:00 Why we're adding factuality as a signal 00:14 Human preference vs. factuality: complementary, not the same 01:06 How this differs from style control 02:32 The math: reviewing the existing Bradley-Terry objective 04:27 Extending the objective to factuality 06:23 The composite objective: weighting human preference + factuality 07:14 Why rankings shift between human preference and factuality 07:34 Defining the factuality label 09:48 Why models aren't penalized for abstaining from claims 13:44 Confidence intervals via M-estimation and asymptotic normality 17:27 Wrap-up: what's next for factuality at Arena