University of Maryland researchers post arXiv paper showing 26.5 to 54 percent mismatch between LLM agents' internal tool necessity detection and actual tool calls

Interesting interpretability paper on tool-using agents. The authors probe hidden states and find the model often recognizes it should call a tool, but fails to actually call one. The mismatch ranges from 26 to 54%, and it concentrates entirely in the cognition-to-action transition, not in cognition itself. In other words, the model usually knows it should call the tool. The internal probe direction is decodable. But the late-layer last-token regime rotates that signal nearly orthogonal to the action it produces. This work tries to predict which interventions will actually work and which will not. Most will blame bad prompting or weak tool-call training, and probably ignore the late-layer geometry. If you have been A/B testing tool-use prompts and getting weird ceilings, this work might offer a good explanation to that behavior. Paper: https://arxiv.org/abs/2605.14038 Learn to build effective AI agents in our academy: https://academy.dair.ai/