Discussions around the HalluHard benchmark show web search often raises hallucination rates in models like Sonnet 5 and Fable 5 during multi-turn exchanges, flipping the expectation that search would ground responses more reliably.
@maksym_andr Only web search matters tbh
💥NEW: We now have Sonnet 5, Fable 5, and Fable 5 + web search on HalluHard. Both models perform better without web search (50.9% and 59.7% hallucination rates), but overall no major breakthroughs here.
For comparison, GLM-5.2 has 74.8%. Everyone is too focused on agentic capabilities, while hallucinations are what matter for ~100% of all users.
Maksym Andriushchenko advocates testing with scaffolds such as Claude Code and Codex CLI to reflect actual usage patterns where search, code execution, and conversation interact.
HalluHard runs three rounds of follow-ups on 950 seed questions, exposing how initial errors grow in domains like legal cases, medical guidelines, and coding.
Users agree web search drives AI hallucinations and express excitement for developing an agentic hallucination benchmark with modern LLM scaffolds to evaluate them properly.
No Digg Deeper questions have been answered for this story yet.
@xeophon i would kindly disagree :) agents can’t use web search for every single statement. that would be too expensive. they need to still rely a lot on their own in-parameter knowledge, especially for long, open-ended outputs.
@xeophon excited to see it!!
@maksym_andr > sounds fair! i think there is room for a proper agentic hallucination benchmark, where LLMs are used with SOTA scaffolds (Claude Code / Codex CLI / OpenCode)
oh boy do I have something in store for you this week
sounds fair! i think there is room for a proper agentic hallucination benchmark, where LLMs are used with SOTA scaffolds (Claude Code / Codex CLI / OpenCode). this would resolve both the web search + linter/pip concerns! also probably content grounding should be made less ambiguous in some cases. nonetheless, i think HalluHard is still pretty valuable as a way to rank hallucination propensity between different models.
(cc @dyfan22 for her opinion!)
@maksym_andr as always, "it depends".
i don't like the coding subset, for example (models have access to linters and pip in all setups these days). for the research questions, i expect models to do a short search
@maksym_andr > sounds fair! i think there is room for a proper agentic hallucination benchmark, where LLMs are used with SOTA scaffolds (Claude Code / Codex CLI / OpenCode)
oh boy do I have something in store for you this week
sounds fair! i think there is room for a proper agentic hallucination benchmark, where LLMs are used with SOTA scaffolds (Claude Code / Codex CLI / OpenCode). this would resolve both the web search + linter/pip concerns! also probably content grounding should be made less ambiguous in some cases. nonetheless, i think HalluHard is still pretty valuable as a way to rank hallucination propensity between different models.
(cc @dyfan22 for her opinion!)
@maksym_andr as always, "it depends".
i don't like the coding subset, for example (models have access to linters and pip in all setups these days). for the research questions, i expect models to do a short search