The framework evaluates cognitive stability in multi-agent workflows.
In agentic pipelines, one LLM can play many roles: writer, reviewer, judge, editor, and more. Does it use the same understanding of a concept across them?
In our new ICML paper, we measure conceptual consistency and test whether more consistent models make fewer mistakes.
Users in the replies are excited to study the consistency dilemma in LLMs for agentic pipelines because operational usefulness in model selection does not guarantee reliability.
No Digg Deeper questions have been answered for this story yet.
Paper: https://arxiv.org/abs/2606.30653 Code + data: http://github.com/MarinaMancoridis/ConsistencyDilemma Joint work with @zhitzig
In a medical setting, we find something surprising: among the models we tested, the more consistent ones were more likely to make clinician-validated mistakes.
We call this the consistency dilemma: when choosing among models, consistency is operationally useful, but doesn’t guarantee reliability.
Much more to do with this metric! We’re excited to study consistency in other settings, and disentangle it from other model-level qualities.
We test 10 frontier models on ~500 concepts across 3 test families. Consistency varies widely across models.
