Report
Hand-crafted regexes are claimed to do heavy lifting in every frontier LLM
The poster says GPT-2's regex had explicit rules for common English contractions and was used in several early-2020s models.
TLDR
In a thread about pretokenizers, a poster claims hand-crafted regexes are part of every frontier LLM and do “heavy lifting.” They point to GPT-2's regex, which they say has explicit rules for common English contractions and was used in several models in the early 2020s. A separate poster calls this “basically” the most time-consuming part of a tokenizer.
Combined views
9.4K
6 Sources, first seen ago
