Priced Guidance is a proposed '20 Questions' test for LLM research ideas

The researchers propose Priced Guidance to assess whether LLMs can generate novel research ideas. Instead of sampling ideas until one matches a future paper’s idea, they have the model play '20 Questions' to recover it, charging each hint by how surprising it is. They say this gives a lower bound on the probability that the LLM would forecast that idea.