Rohan Anil draws on his Gemini pretraining work to argue that open models would train faster if the industry agreed on core components, starting with the tokenizers that turn text into model inputs. He flags the current patchwork of tokenizer choices as lacking clear technical justification and positions early standardization as a practical way to accelerate collective progress.
It will be exciting times when we start collaborating on standardization around every part of the training tech stack starting from the tokenizer.
It’s not that clear why open models use slightly different tokenizers. That would be a good accelerant.
Replies in the thread note that efficiency gains from consistent tokenizers grow more important precisely because labs now train on increasingly different data sources.
Beyond the tokenizer starting point, no proposed standards, timelines, or participating organizations have been outlined, leaving the scope of any collaboration open.
No Digg Deeper questions have been answered for this story yet.
My personal expectations is there are a few algorithmic moves to give hardware + model codesign moats.
@_arohan_ Training efficiency (even more significant now that labs are not at all using the same data sources).
It will be exciting times when we start collaborating on standardization around every part of the training tech stack starting from the tokenizer.
It’s not that clear why open models use slightly different tokenizers. That would be a good accelerant.
@Dorialexander Its not that much and particularly given many of them >128k tbh
@_arohan_ still tight compression for multilingual.
@_arohan_ honestly, i want only English model, drop other tokens and focus only one language with frontier capability.
@_arohan_ yeah but inference at scale for a month flips that. training is one-time, serving is forever.