Quail reportedly classifies documents with LLMs about 1.8x faster than baselines

A blogger says Quail classifies documents with LLMs about 1.8x faster than baselines. They say generating labels as free text requires memory-bound, token-by-token decoding that leaves GPUs mostly idle. A user sharing the post attributes the claimed speedup to using a trie instead of free-text generation and argues that structured decoding is better suited to fixed label sets.