Baichuan AI Tokenizer Embeds CCP Political Terms and Propaganda Phrases
Reactions from ranked influencers
2 posts
oh you're not dissing the ccp? you must be an agent. no, i'm simply literate.
This is interesting because: https://www.reuters.com/technology/chinas-baichuan-zhipu-ai-launch-ai-models-after-receiving-approval-2023-08-31/ and these approvals include passing: https://digichina.stanford.edu/work/translation-measures-for-the-management-of-generative-artificial-intelligence-services-draft-for-comment-april-2023/ particularly, Article 4:
lord give me the confidence of a man who can so confidently assert that the CCP would have no understanding of open-source risks/have strategic blindness towards "AGI" à la yann-lecuny vibes outside of masterfully insulting both, it is a Very Bad Take
I don't know why I'm spending my Friday night looking at tokens, but here we are. The Baichuan tokenizer has a 64k vocab size, of which ~28k contains Chinese characters, and ~1.5k are >= 3 characters. Perhaps unsurprisingly, we find: "Epidemic prevention" "Coronavirus disease" "Committee" "Xi Jinping" "Coronavirus" "Nucleic acid amplification testing" "New coronary virus" "wear mask" "Communist Party" "People's Republic of China" "Communist Party of China" "General Secretary Xi Jinping" "Copyright belongs to the original author" "The copyright belongs to the original author" (The list of ~1.5k Chinese tokens can be found here: https://gist.githubusercontent.com/suchenzang/2882c6dd58fc938cae98bac35150973e/raw/cec97b70fae622922d917d2bdfc83f680b53d133/baichuan_vocab_with_long_chinese_chars.txt)