CATT benchmark tests why AI attribution methods miss training documents

CATT's researchers at Northeastern and MIT report that attribution methods often fail to find documents known to have taught a model a fact. They fine-tuned models on fictitious facts to establish which documents taught each answer. In their diagnostic benchmark, scoring each document's top 10 tokens and using negative log-odds gradients improved recall of the correct documents by 35 to 72 percentage points across multiple baselines.