UChicago professor flags language model probability mismatch

Suproteem Sarkar, Assistant Professor at UChicago, posted an example in which a language model is asked the probability of rain tomorrow and then the probability it will not rain. The post states that the two probabilities should sum to one regardless of the outcome yet shows a case where the model generates 0.7 for rain and 0.2 for no rain. The tweet links to a flowchart diagram titled "Can you arbitrage a language model? Evaluating probabilistic coherence of model forecasts" that depicts two parallel rows of four connected boxes.