New Preprint Finds LLMs Fail to Reliably Use Uncertainty Markers for Confidence

(15/n) If you're interested in LLM uncertainty, reliability, or human-AI interaction, please check out our paper and analysis framework! We'd love feedback or questions :))

📄 Paper: http://arxiv.org/pdf/2605.28778 🔗 Github: http://github.com/yale-nlp/marker_internal_confidence

(7/n) We compute & analyze MICs for 13 leading open- and closed-source LLMs across 11 diverse tasks spanning knowledge-intensive QA, math, science, hallucination detection, and more. This enables fully 🎯 model-centric evaluation of LLMs’ linguistic confidence.

(2/n) We know LLMs often say things 💪 assertively when they're actually ❌ wrong. But here's a subtler question: when models 𝘥𝘰 hedge — does “I think" mean something different internally from "probably"? And do these markers carry stable, distinguishable confidence signals?

(5/n) We formalize 🎯 𝗠𝗮𝗿𝗸𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗻𝗮𝗹 𝗖𝗼𝗻𝗳𝗶𝗱𝗲𝗻𝗰𝗲 (𝗠𝗜𝗖): the average internal confidence a model associates with a given epistemic marker (e.g., "I think," "might") in a given task domain. We then ask: is this association stable and meaningful?

(3/n) Prior work shows LLMs' hedge use does not match 🧑 human interpretation of these words. But humans also vary in how we interpret hedges! This raises a fundamental question: do models apply their own linguistic confidence framework—even if different from ours?

(14/n) Overall, we provide critical, complementary evidence to prior work that LLMs are faithfully miscalibrated even when evaluated entirely on their own terms, pointing to a fundamental gap between LLMs' surface level linguistic choices and internal confidence signals.

(6/n) 🔍 We propose 𝟳 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 to evaluate MIC stability & consistency within and across tasks. We study: 📏 How well do LLMs differentiate markers by MIC? 📊 Are marker meanings and rankings consistent across tasks? 🧪 How well do MICs predict accuracy and faithfulness?

(4/n) In this work, we test that possibility directly. Even if a model's hedge use doesn't match human intuition, its markers can still be meaningful if it 𝘤𝘰𝘯𝘴𝘪𝘴𝘵𝘦𝘯𝘵𝘭𝘺 maps them to specific internal confidence levels.

⚠️ 𝗞𝗲𝘆 𝗙𝗶𝗻𝗱𝗶𝗻𝗴 #𝟱: MICs somewhat track 𝘢𝘤𝘤𝘶𝘳𝘢𝘤𝘺 and 𝘧𝘢𝘪𝘵𝘩𝘧𝘶𝘭𝘯𝘦𝘴𝘴 of expressed uncertainty. High-MIC markers emerge as a primary driver of faithful miscalibration — models use confident-sounding language regardless of internal uncertainty.

⚠️ 𝗞𝗲𝘆 𝗙𝗶𝗻𝗱𝗶𝗻𝗴 #𝟭: Models barely distinguish hedges by internal confidence. MIC values are highly concentrated across markers, even when models' overall confidence spans the full 0–1 range.

(8/n) The results? Even when evaluated on their own terms, LLMs struggle to use epistemic markers consistently & reliably 😬

⚠️ 𝗞𝗲𝘆 𝗙𝗶𝗻𝗱𝗶𝗻𝗴 #𝟰: Larger models 📈 show better MIC stability & ranking consistency across tasks. But they also 📉 discriminate poorly between hedges. Metacognitive prompting to improve human-aligned hedge use and faithfulness doesn't fix the underlying structure.

⚠️ 𝗞𝗲𝘆 𝗙𝗶𝗻𝗱𝗶𝗻𝗴 #𝟯: Marker rankings hold — but for the wrong reason. ❌ Most models preserve a fairly stable ordering of hedges across tasks (e.g., "highly likely" > "might"). But remove the distinction between hedging vs. not hedging, and the ranking largely collapses.

⚠️ 𝗞𝗲𝘆 𝗙𝗶𝗻𝗱𝗶𝗻𝗴 #𝟮: MICs are sensitive to distribution shift. Cross-domain MIC consistency is substantially weaker than in-domain for all models. What "likely" internally means on science QA ≠ what it means on knowledge-intensive QA.

(16/n) Shoutout to my co-author @armancohan!