• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
AI

New tokenization framework announced

A user touts support for multiple languages, multithread scaling and a minimal package size, linking to a Hugging Face blog.

Hugging FaceHF
clem 🤗C🤗
LysandreLY
8 Sources, 20d ago, first seen 20d ago

TLDR

A post announces a new tokenization framework and links to a Hugging Face blog. The author calls it “state of the art,” highlighting support for multiple languages, multithread scaling and a minimal package size.

Combined views

50K

8 Sources, first seen 20d ago

599 likes21 comments386 saves79 reposts

Combined views

50K

8 Sources, first seen 20d ago

599 likes21 comments386 saves79 reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Featured Source

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

8 Sources

Arthur Zucker@art_zuckerHappy to officially bring you the new SOTA `tokenization` library. We focused on all languages, multi-thread scaling, minimal package size and memory usage. https://huggingface-tokenizers-v1.static.hf.space/index.html We're thankful for the players of this ecosystem that have pushed us to give the best we could!20d
Lewis Tunstall @ COLM 🌉@_lewtunRT @art_zucker: Happy to officially bring you the new SOTA `tokenization` library. We focused on all languages, multi-thread scaling, minim…20d
Lysandre@LysandreJikTokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization, rivaling the best in the ecosystem. Scaling across threads, much better latency, less memory consumed, and microscopic crate size. https://huggingface.co/blog/tokenizers-v120d
Pedro Cuenca@pcuenqtokenizers v1 (Rust): 3 to 30 times faster, opt-out tokenization training crate, lots of clever algorithms and optimizations, extensive benchmarks. Coming soon to transformers. https://huggingface.co/blog/tokenizers-v120d
Aritra 🤗@ariG23498Wake up babe! State of the art tokenization framework just dropped. > multiple language support > multi-thread scaling > minimal package size Read: https://huggingface.co/blog/tokenizers-v120d
Hugging Face@huggingfaceRT @LysandreJik: Tokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization,…20d
clem 🤗@ClementDelangueRT @ariG23498: Wake up babe! State of the art tokenization framework just dropped. > multiple language support > multi-thread scaling > m…20d
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    8 Sources

    Arthur Zucker@art_zuckerHappy to officially bring you the new SOTA `tokenization` library. We focused on all languages, multi-thread scaling, minimal package size and memory usage. https://huggingface-tokenizers-v1.static.hf.space/index.html We're thankful for the players of this ecosystem that have pushed us to give the best we could!20d
    Lewis Tunstall @ COLM 🌉@_lewtunRT @art_zucker: Happy to officially bring you the new SOTA `tokenization` library. We focused on all languages, multi-thread scaling, minim…20d
    Lysandre@LysandreJikTokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization, rivaling the best in the ecosystem. Scaling across threads, much better latency, less memory consumed, and microscopic crate size. https://huggingface.co/blog/tokenizers-v120d
    Pedro Cuenca@pcuenqtokenizers v1 (Rust): 3 to 30 times faster, opt-out tokenization training crate, lots of clever algorithms and optimizations, extensive benchmarks. Coming soon to transformers. https://huggingface.co/blog/tokenizers-v120d
    Aritra 🤗@ariG23498Wake up babe! State of the art tokenization framework just dropped. > multiple language support > multi-thread scaling > minimal package size Read: https://huggingface.co/blog/tokenizers-v120d
    Hugging Face@huggingfaceRT @LysandreJik: Tokenizers v1's first release candidate is out 🔥 absolutely huge release across the board: Up to 30x faster tokenization,…20d
    clem 🤗@ClementDelangueRT @ariG23498: Wake up babe! State of the art tokenization framework just dropped. > multiple language support > multi-thread scaling > m…20d
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet