Lewis Tunstall#958
Deep dive into FNS: building a tokenizer that chunks text efficiently but has character level resolution!
FNS augments the loss with character level signal at training time while at inference time you can decode single characters.
Deep dive here: https://huggingface.co/spaces/HuggingFaceBio/carbon-tokenization
Users are praising the FNS Tokenizer as a super clever approach for efficient DNA chunking with character-level resolution.
@ChainZenit thanks! @QiuyiLi2‘s work!
@lvwerra nono, can't do without you 😊
@lvwerra character level resolution is nice but id wanna see memory impact vs subword tokenizers
@lvwerra that’s actually a super clever way to handle tokenization.
@lvwerra wait so it has character level precision but still chunks efficiently?
is this the tokenizer that finally kills the subword tradeoffs?
@lvwerra @danaaubakir 🔥🔥🔥
@rugbist_ it depends a lot on your application. this works well on DNA where you only have 4 characters, so you can easily create all combinations of tokens to a certain lenght. natural language is a bit trickier.
@aeronxbt The tokenizer does split into chunks of 6 characters, so you get all the advantages of the tokenizer.
Lewis Tunstall#958Deep dive into FNS: building a tokenizer that chunks text efficiently but has character level resolution!
FNS augments the loss with character level signal at training time while at inference time you can decode single characters.
Deep dive here: https://huggingface.co/spaces/HuggingFaceBio/carbon-tokenization