Commentary on X
@sakurayukiai @woosuk_k @lightseekorg Exactly. With full attention, sliding windows, and varied reuse patterns, it’s no longer just allocation: admission, placement, eviction, and scheduling are coupled. That’s why TPS alone is thin—tail latency and eviction churn tell more of the story.
The Conversation on X
2 postsKV cache management is arguably one of the hardest problems in LLM inference, and advancing it requires the community to learn from and build on one another’s work. Great to see that happening. Amazing work from @lightseekorg on adopting and evolving vLLM's hybrid memory allocator! The Jenga paper from @ChenZha62999224 is especially relevant today as attention architectures become increasingly complex.
TokenSpeed scheduler originally used a Radix Tree–based memory pool. As new attention patterns became more complex, we deprecated the Radix Tree–based design and implementation, and rebuilt the memory manager with a flat, block-based KV cache architecture. TokenSpeed now uses a single flat paged pool with heterogeneous views, making it easier to support different attention mechanisms. Similar approaches have also been explored in the community, including @vllm_project's Jenga and LMDeploy’s TurboMind. This redesign happened around the release of TML’s Inkling, so we were able to support Inkling from day one with the new architecture. We are excited to keep building with the open-source community. 👇