A 'tokenizer tax' could make AI costlier in some languages

Rest of World reports a Coalition for Language Equity in AI Research study found a gap even among the 20 most popular internet languages: the most expensive used 1.5 to 5 times more tokens than the cheapest. Researchers say higher token counts can raise costs, slow responses and reduce room for conversation history. Languages written in scripts tokenizers rarely saw, including Burmese and Tibetan, faced the greatest disadvantages across nearly all models analyzed.