Better AI-agent scores can mask worse results on skill-retrieval tasks, a post warns

A user summarizing “Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents” argues that comparing tasks where an agent retrieved skills with tasks where it did not can be misleading. If the agent retrieves skills on easier problems, retrieval can look helpful even when it does nothing or makes the answer worse. According to the summary, the paper’s proposed fix, RAE, compares performance on the same task with and without skill access whenever retrieval occurs.