/Tech1h ago

Anthropic Tokenization Overhead Surges for Non-English Languages Versus OpenAI

18042.2K

Original post

Susan Zhang@suchenzang#84inTech

h/t:

The non-English tax is real.

Sutton's Bitter Lesson, translated across languages and normalized to OpenAI English token count:

Hindi: OpenAI 1.37×, Anthropic 3.24× Arabic: OpenAI 1.31×, Anthropic 2.86× Chinese: OpenAI 1.15×, Anthropic 1.71×

Claude’s tokenizer charges a much higher linguistic tax.

9:08 AM · Jun 15, 2026 · 1.9K Views

Sentiment

Sentiment building, check back later.

Cluster Engagement

Posts from X

Most Activity

VIEWS291LIKES3

@deliprao we're simply getting a masterclass in the delta between a technical problem, market manipulation, and the legal/governmental powers that be

@suchenzang something about glasshouses and stones

1h29130

umumu@umi33563

@suchenzang Wait huh dfuk what??? How did they manage to be almost 2x worse than English baseline on French?

1h1