Same ideas, different verdicts: AI novelty judges have a prompt problem, study finds
Researchers report that a prompt change can flip novelty verdicts on more than half of unchanged research-idea pairs.
TLDR
The ideas stayed put; the verdicts didn’t. In a study of six LLM judges, researchers report that a prompt change can alter verdicts on more than half of unchanged idea pairs and shift pairwise accuracy by over 50 percentage points. The benchmark relies on selected reviewer judgments and weakly labeled generated ideas. It’s a warning about measuring AI research novelty—not proof that AI can’t discover anything.
Combined views
2 Sources, first seen ago

Comments