Researchers Discuss LLMs and Shannon Text Redundancy
AI researchers share posts on Shannon entropy and LLM text compression.
Ahmad Beirami retweeted a post by docmilanfar referencing Claude Shannon's estimate from over 75 years ago that English text is roughly 75 percent redundant. The post and a generated headline claim modern LLMs compress English below Shannon's one bit per character limit. Christian Szegedy quoted the same thread and stated that optimizing for language compression produced current AI capabilities. The packet records these public posts on X without independent confirmation of the compression results.
Combined views
19.6K
2 posts, first seen 2d ago
Reactions
2 postsPM 
Peyman Milanfar@docmilanfarOver 75 yrs ago, Shannon estimated that English is ~75% redundant. When you consider the long-range context of a phrase or sentence, the entropy is no more than ~1 bit per letter. It wasn't until very relatively recently that text encoders achieving this rate were created. In…
- likes: 1K
- replies: 19
- bookmarks: 466
- reposts: 97
CS Christian Szegedy@ChrSzegedyIMO, the even more profound and surprising result is that optimizing for compression of language got us to the current level of AI.
- likes: 209
- replies: 10
- bookmarks: 81
- reposts: 16
