Bill Yuchen Lin of xAI warns CJK tokenization remains an unsolved challenge, citing persistent glitches in newer models like Fable

@billyuchenlin xswl

@abdurrahmanregi haha we also saw this in GPT models when we were investigating the issue. some tokens are just too powerful.

@billyuchenlin gemma 4 E2B and E4B still does 'уточнить' 😂

One of the beauties of life, is it's dynamicity! Humans have different cultures, and now, our machines are learning it too.

While humans aren't perfect, machines exhibit this characteristic too, however, but human and machine can get better,

Through continuous training and self development. Kudos to you and your team.

@billyuchenlin Reward shaping and top/min-p 🙏