Whatever AI sceptics say, LLMs really can reason. They're not just doing an imitation that looks like reasoning, it's the real deal.
But even though they are able to reason, sometimes they won't! If you ask an LLM a question it can't answer, sometimes it will just try to imitate reasoning without doing it.
The chain of thought looks basically indistinguishable from actual reasoning. But under the hood something very different is going on.
@TrentonBricken talked with me about what work on circuits inside LLMs has revealed: