Researchers introduce TokenGate to prune tokens during long-context prefill

TokenGate is a learned component designed to skip less important tokens as language models process long inputs. In their paper, Liana Patel and co-authors report 1.4 to 11.3 times faster time-to-first-token and 1.5 to 3.4 times lower peak memory across diverse tasks, with no accuracy degradation. Those figures are the authors’ reported evaluation results.