Three purported myths in language-model tokenization research
A post quotes a research excerpt that calls tokenization fundamental to modern language models: a tokenizer determines how data appears before a model sees it.
TLDR
A post shares an excerpt from a work that says it identifies and challenges three myths in tokenization research. The excerpt says a tokenizer determines how data appears before a language model sees it.
Combined views
3.5K
1 Source, first seen 3h ago
29 likes