Language Compression Optimization Fuels Current AI
Researchers discuss Shannon's entropy estimates and their link to LLM capabilities.
Christian Szegedy posted that optimizing for compression of language produced the current level of AI. Peyman Milanfar wrote that Claude Shannon estimated English text is roughly 75 percent redundant, with long-range entropy at most 1 bit per letter. Milanfar added that text encoders reaching this rate appeared only recently and that modern LLMs now predict the next character at this level.
Combined views
81.6K
2 posts, first seen 8d ago
Language Compression Optimization Fuels Current AI
Researchers discuss Shannon's entropy estimates and their link to LLM capabilities.
Christian Szegedy posted that optimizing for compression of language produced the current level of AI. Peyman Milanfar wrote that Claude Shannon estimated English text is roughly 75 percent redundant, with long-range entropy at most 1 bit per letter. Milanfar added that text encoders reaching this rate appeared only recently and that modern LLMs now predict the next character at this level.
Reactions
PM 
Peyman Milanfar@docmilanfarOver 75 yrs ago, Shannon estimated that English is ~75% redundant. When you consider the long-range context of a phrase or sentence, the entropy is no more than ~1 bit per letter. It wasn't until very relatively recently that text encoders achieving this rate were created. In…
- likes: 1K
- replies: 19
- bookmarks: 466
- reposts: 97
CS Christian Szegedy@ChrSzegedyIMO, the even more profound and surprising result is that optimizing for compression of language got us to the current level of AI. https://twitter.com/docmilanfar/status/2092767297983406086
- likes: 211
- replies: 10
- bookmarks: 81
- reposts: 17