✍️ Good forecasts often come with rationales—written explanations of the reasoning behind a number.
In our studies, we've collected millions of words of rationales, where forecasters explain their logic, cite evidence, and weigh competing considerations.
But there are crucial things we don't know about rationales. For example—which features of a rationale are good predictors of forecasting accuracy?
Thank you to the study's authors: Chris Karvetski, @sicong_huang, @simas_kucinskas, Nadja Flechner, Jingyu Hu, @PTetlock, and @EzraKarger
Read more on our Substack: https://forecastingresearch.substack.com/p/can-you-judge-a-forecast-by-its-rationale
Read the full working paper on SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6862818
Why was this?
As you can see below, human ratings correlate strongly with rationale length, but length is essentially uncorrelated to forecast-level accuracy.
Human raters weren’t wrong directionally, but they appeared to place undue weight on some features, such as underweighting "red flags" like extreme confidence.