Reactions from ranked influencers
2 posts
Gigatoken is absurdly fast! Thanks to Marcel, I will (hopefully) never have to wait for tokenization again. It turns out that even mature components like tokenizers have order-of-magnitude improvements left if you get close to the hardware and write state machines.
Introducing the world's fastest tokenizer implementation, Gigatoken! Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines. These baselines are already multithreaded Rust implementations! 🧵
Introducing the world's fastest tokenizer implementation, Gigatoken! Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines. These baselines are already multithreaded Rust implementations! 🧵