Perplexity open-sources a rebuilt Unigram tokenizer that reduces CPU utilization by 5x to 6x

Every millisecond matters. We’re open sourcing the tokenizer we built and deployed on production; that’s far efficient than huggingface and sentencepiece.

At production input lengths, the encoder cuts p50 latency by roughly 5× vs. HuggingFace tokenizers, 2× vs. SentencePiece C++, and 1.5× vs. IREE C.

At 514 tokens, it runs in 63 µs with zero heap allocations.

The work targets XLM-RoBERTa’s 250K-token Unigram vocabulary, commonly used for ranking and retrieval.

The encoder produces the same tokens as the reference implementation, but avoids rebuilding strings and chasing hash maps while deciding how text should be split.

Read more about improving Unigram tokenizer CPU performance on our blog:

https://research.perplexity.ai/articles/improving-unigram-tokenizer-cpu-performance

@AravSrinivas Super cool

@perplexity_ai Future perplexity Korea B2C "ME"

@perplexity_ai this is the kind of latency work that actually matters.

once the model pass is single-digit ms, all the boring CPU pieces stop being boring.

@perplexity_ai Rebuilding a tokenizer to slash CPU utilization by 5-6x is an incredible engineering flex. But it’s wild that we optimize single-digit milliseconds under the hood, yet the desktop user interface remains locked in a rigid, hardcoded container surrounded by dead space.

@perplexity_ai Good signal for where inference optimization is moving.

Once rerankers/embedders run in single digit ms on GPU, CPU tokenization and heap allocation stop being background noise.

The model is not always the bottleneck. The path into the model starts to matter.

@perplexity_ai 5-6x CPU reduction is a real lever here: once rerankers are single-digit ms on GPU, tokenization stops being plumbing and starts showing up in p95.

@perplexity_ai This is actually cool

@perplexity_ai zero heap allocations at 63µs is the thing. cpu tokenization becoming the gc pressure point in high-throughput inference is what creeps up on you.

@perplexity_ai 5-6x CPU win on tokenization is the kind of unsexy infra optimization most teams skip until latency budgets blow up. Glad it is open. Reranker stacks badly need this.

@perplexity_ai 5-6x just on tokenization is wild. easy to forget the CPU side once the model itself is single-digit ms on GPU.

@AravSrinivas Amazing work @aravsrinivas! 5-6x lower CPU utilization on tokenization is huge for production inference.

Every ms counts when your rerankers/embedders are already blazing fast on GPU. Will definitely try pplx-unigram from pplx-garden 🔥

@perplexity_ai At 100ms inference, tokenization is rounding error. At 8ms GPU it's 20% of wall time. Batching overhead and HTTP fan-out are probably next in line. The optimization queue for fast retrieval keeps moving up the stack.

@perplexity_ai love that the actual rewrite is open-sourced and not just the benchmark number

@perplexity_ai @Presidentlin

@perplexity_ai GPU inference is fast now. Fast enough that CPU tokenization is suddenly the bottleneck.

Perplexity rebuilt Unigram from scratch to fix it.

@AravSrinivas Your tokenizer won't move the needle unless it actually ships.