Potential AI gains in hard-to-verify reasoning may be hard to spot

A post speculates that AI models may be improving at “fuzzy” reasoning, such as conceptual thinking, scientific research judgment and strategy, in ways that are difficult to recognize. The author suggests models might struggle to explain novel insights, while people have less agreement about how to judge them than they do for math proofs. Better ways of eliciting those abilities, such as self-play or distillation, could reveal progress.