Hand-crafted regexes are claimed to do heavy lifting in every frontier LLM

In a thread about pretokenizers, a poster claims hand-crafted regexes are part of every frontier LLM and do “heavy lifting.” They point to GPT-2's regex, which they say has explicit rules for common English contractions and was used in several models in the early 2020s. A separate poster calls this “basically” the most time-consuming part of a tokenizer.