Rohan Anil draws on his Gemini pretraining work to argue that open models would train faster if the industry agreed on core components, starting with the tokenizers that turn text into model inputs. He flags the current patchwork of tokenizer choices as lacking clear technical justification and positions early standardization as a practical way to accelerate collective progress.
It will be exciting times when we start collaborating on standardization around every part of the training tech stack starting from the tokenizer.
It’s not that clear why open models use slightly different tokenizers. That would be a good accelerant.
Replies in the thread note that efficiency gains from consistent tokenizers grow more important precisely because labs now train on increasingly different data sources.
Beyond the tokenizer starting point, no proposed standards, timelines, or participating organizations have been outlined, leaving the scope of any collaboration open.
Users welcome standardizing the AI training tech stack starting with tokenizers because it could unlock easier collaboration and major efficiency gains like 2x improvements.
No Digg Deeper questions have been answered for this story yet.
@_arohan_ Training efficiency (even more significant now that labs are not at all using the same data sources).
It will be exciting times when we start collaborating on standardization around every part of the training tech stack starting from the tokenizer.
It’s not that clear why open models use slightly different tokenizers. That would be a good accelerant.
@Dorialexander Its not that much and particularly given many of them >128k tbh
My personal expectations is there are a few algorithmic moves to give hardware + model codesign moats.
was workin on a tokenizer concept last night - intentionally mispelling by the llm to reduce tokens on non-value words, and experimenting with rhyme condensation. -ther, -ing, seeing how far you can take compressing the likes, and only using the differences. like caseops but on steroids and for people who dont mind reading mispelled words. whats the pareto there, and do you get more value tokens in by using linguistic drift? I didn't look it up before starting because that ruins all the fun
@_arohan_ still tight compression for multilingual.
@_arohan_ honestly, i want only English model, drop other tokens and focus only one language with frontier capability.
@_arohan_ yeah but inference at scale for a month flips that. training is one-time, serving is forever.
@_arohan_ Standardization will enable an easy 2x improvement across the stack at minimum. Probably 10x in the long run
@_arohan_ I'd love to be wrong but this feels like an "if everyone would just" situation
@_arohan_ Different user groups, different priorities. Wouldn't any non-english lab want to optimize tokens for their own consumers n get cheaper inference compared to others.
@_arohan_ tokenizers feel like an underrated place to collaborate
@_arohan_ SlipGram is the name mmmk