New Paper Shows Frontier Models Struggle Evaluating Grade-School Math Reasoning

💡 Why does this matter?

As people increasingly use frontier models to write research papers, produce proof attempts, or generate persuasive arguments, this gap between producing arguments and vigilantly assessing them becomes a societal vulnerability, not just a technical one: If AI can produce plausible-sounding reasoning at scale, but not help us weed out what’s actually invalid, our ability to do science and make sense of the world may be significantly harmed.

How might we address this gap? In The Enigma of Reason (2017) — one of the inspirations for our work — the cognitive scientists Hugo Mercier and Dan Sperber suggest that human reasoning evolved via social incentives, and that being critical evaluators allows us to gain the benefits of others’ thinking while avoiding being misled. In contrast, AI models are trained to reason in isolation, resulting in very different incentives. By learning from human cognition, we could potentially reduce the production-evaluation gap.

(*Results on Fable 5 are freshly run, and not yet included in our paper.)

🤝 Joint work with Teresa Yeo (@aseretys), Armando Solar-Lezama, and Tan Zhi-Xuan (@xuanalogue).

📄 Paper: https://arxiv.org/abs/2606.01462. #LLMs #LRMs #Reasoning #AI4MATH #CogSci

Of course, CoTs need not be faithful to what to the evaluator models are doing under-the-hood, so we also find mechanistic evidence of the bias at work: :

🔸Linear Probes: Using probes trained on LRM activations, we find that these activations encode some representation of valid reasoning. However, we also find that these internal representations get corrupted and dynamically overridden by the presence of a valid final answer in VAIR solutions.

🔸Causal Patching: By swapping the hidden states of a valid answer token with an invalid one, we can causally flip the model's validity verdict and activations —- once the model detects that the final answer is wrong, it is no longer inclined to judge invalid reasoning steps as valid

Together, these results demonstrate the operation of answer confirmation bias at inference time. But is this bias ultimately the result of outcome-focused incentives at training time, as we hypothesize? We plan to investigate this in future work.

🔍 What is VAIR?

To isolate a model's ability to evaluate reasoning — and prevent it from using its ability to produce reasoning to solve our tasks — we took gold-standard math solutions and injected trivial reasoning flaws (e.g., deleting an essential step, shuffling the order) while keeping the final answer perfectly valid.

If a model actually reads and evaluates the steps, it should detect the flaw and judge the solution as containing invalid reasoning. But if the model just solves the problem on its own and verifies that the final answer is correct, it will incorrectly judge the solution as valid.

When we subject frontier LRMs to our VAIR dataset, their evaluations consistently break down. Despite near-perfect solution accuracy, these models suffer massive performance drops when evaluating VAIR solutions (as low as 48%).

For context, human reasoners are significantly more robust at our task; we found that humans are only 6% worse at grading these tricky problems than they are at solving them —- consistent with research in cognitive science that people are incentivized to be epistemically vigilant against misleading arguments.

🧠 Why does this happen?

We trace this back to an Answer Confirmation Bias.

Because LRMs are predominantly trained via outcome-based reinforcement learning, they are incentivized to produce correct answers, not to validate reasoning. One hypothesis then, is that this training ends up biasing LRMs to rely on answer validity when evaluating reasoning.

Indeed, when we inspect the verbalized “chain-of-thought” (CoT) produced by our evaluator models, we find many symptoms of answer confirmation bias: When evaluating a VAIR solution, CoT wording indicates that LRMs often (i) independently solve the problem by themselves (ii) verify that the solution’s answer is valid, then (iii)overlook reasoning flaws or fabricate rationalizations for the solution’s broken logic.