@svpino 87% claim accuracy still means half of responses have an error? Brutal math for healthcare AI. #HealthTech #AIEthics Cc
@svpino half of responses having at least one wrong claim feels generous tbh
@svpino coin flip every time you ask it something lol
That's crazy, I didn't understand this until you explained it @svpino
Many users criticized the new factuality metric exposing false claims in half of AI model responses, calling the error rate brutal for healthcare and likening reliability to a coin flip.
Based on 3 visible X reactions from 9 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@svpino 87% claim accuracy still means half of responses have an error? Brutal math for healthcare AI. #HealthTech #AIEthics Cc
@svpino half of responses having at least one wrong claim feels generous tbh
@svpino coin flip every time you ask it something lol
That's crazy, I didn't understand this until you explained it @svpino
Models make about 5 web-verifiable claims per response in Text Arena, and 87% of those claims check out. 0.87^5 = 0.50. This means that roughly half of the responses contain at least one false claim! Search Arena is even worse: nearly 10 claims per response at 89% true. 0.89^10 = 0.31, so about 7/10 responses have something wrong with them. Obviously, claims within a response aren't independent, so treat the exact numbers as rough, but the idea is there. "Factuality" is a new way to rank models that takes this into account.
Introducing factuality in the Arena: a new ranking of models according to a weighted combination of human preference and factuality. Model rankings are now viewable according to a weighted combination of human preference and factuality. Factuality is live in our Text and Search Arenas as a non-default toggle. We audit model responses by randomly sampling battles and extracting web-verifiable claims. We then verify these claims and compare the average correctness between model responses. To power these rankings, we’ve labeled over 2 million claims made by LLMs in real-world conversations, 1.3+ million from Text Arena, and 700k+ from Search Arena. Notable highlights with factuality enabled in the Text Arena: - Claude Fable 5 moves down slightly to spot #2 - GPT-5.5 saw the largest increase, moving up 13 spots into the #7 spot - Muse Spark dropped the most from #7 to #20 (-13pt) By labs, Meta saw the largest drop from #2 to #5, while Anthropic overall held the #1 spot. Looking at only open model providers, Xiaomi saw the largest improvement, jumping from #9 to #6. Learn more about the findings and methodology in this thread.
That's crazy, I didn't understand this until you explained it @svpino
Models make about 5 web-verifiable claims per response in Text Arena, and 87% of those claims check out. 0.87^5 = 0.50. This means that roughly half of the responses contain at least one false claim! Search Arena is even worse: nearly 10 claims per response at 89% true. 0.89^10 = 0.31, so about 7/10 responses have something wrong with them. Obviously, claims within a response aren't independent, so treat the exact numbers as rough, but the idea is there. "Factuality" is a new way to rank models that takes this into account.
Many users criticized the new factuality metric exposing false claims in half of AI model responses, calling the error rate brutal for healthcare and likening reliability to a coin flip.
Based on 3 visible X reactions from 9 accounts; directional sample.
Ask a question below.
Published answers will appear here.
