Retest of 50 LLM findings classifies 39 as expired or weakened

Researchers revisited 50 widely cited findings about large language models across 20 models in a study shared Nov. 2 by oq_35. They classified 22 findings as expired and 17 as weakened. Eight persisted, and three strengthened. The verdicts describe fitted effects as of Sept. 30, 2026.

To track changes, the researchers say they ran the same setup for each finding across models released from July 2024 to September 2026 by eight developers, then fitted trends against model release dates. They placed a finding in the expiration zone when its fitted effect fell to one-fifth or less of the effect in the original paper. A finding could meet that cutoff even if some individual models still showed a strong effect.

The expired findings listed on the project page include fabricated citations and miscounted letters. The researchers say recent models rarely fabricated citations when given prompts from an earlier study, though some models still did. The three strengthened findings concerned judges favoring their own writing, praise following a user's preference and nonuniform random-number generation.