AI assistants may answer fewer questions when unused tools are available
DAIR.AI says a paper found that answer rates on questions needing no tools fell from 98.2% to 63.5% across six models when a related tool was merely available.
TLDR
According to DAIR.AI’s summary of the paper, a related tool’s availability reduced answer rates even when the tool went unused. The benchmark covered 500 query pairs across 10 domains, each with a tool-unavailable control.
The summary says a preceding tool call recovered some lost answers but caused new losses. A one-sentence system instruction explaining the tool’s purpose recovered up to 45.6 points.
Combined views
5.1K
1 Source, first seen 15d ago