LLMs Produce Narrower Outputs Than Training Data

Rohan Paul posted that LLMs learn a narrower set of answers than their training data, even when sampling rather than using greedy decoding. The paper compares model continuations with training continuations for identical prefixes and measures output variety after accounting for the prompt. It examines OLMo, Pythia, and GPT-Neo. The tweet includes a screenshot of the academic paper as its attachment.