/ai

UMBP reportedly cuts extra KV-cache copies in SGLang

SemiAnalysis says UMBP's integration in SGLang replaces HiCache's intermediate buffer with a direct path between high-bandwidth memory and DRAM, using a shared pool per node.

SE

2 Sources, 2d ago, first seen 2d ago

TLDR

SemiAnalysis describes HiCache's intermediate host-memory buffer as requiring an extra copy whenever key-value (KV) cache data is loaded or offloaded, with fetches that are not pipelined with computation. It says UMBP provides a direct, layer-wise pipelined path that removes the intermediate copy and stall. The shared pool also deduplicates replicated KV data and allows global placement and eviction decisions across data-parallel ranks and instances, according to SemiAnalysis. The publisher credits AMD's SGLang team, its MoRI (Modular RDMA Interface) library and UMBP's integration for the improvements.

Today's Rank

—

Not ranked yet

Combined views

16.4K

2 Sources, first seen 2d ago

2 Sources

@SemiAnalysis_HiCache sits between HBM and the external DRAM KV store as a per-rank host-side L2 buffer, so every load and offload takes an extra copy, the L3-to-L2 fetch is not pipelined with compute, and under MLA with TP each rank replicates the same KV across PCIe and host memory while making eviction decisions on local information only. UMBP through the SGLang KVCache Store Linker replaces that with a direct, layer-wise pipelined HBM-to-DRAM path into a single shared per-node pool, which removes the intermediate copy and stall, deduplicates the replicated KV, and lets placement and eviction be decided globally across DP ranks and instances. (3/3)@SemiAnalysis_HiCache sits between HBM and the external DRAM KV store as a per-rank host-side L2 buffer, so every load and offload takes an extra copy, the L3-to-L2 fetch is not pipelined with compute, and under MLA with TP each rank replicates the same KV across PCIe and host memory while making eviction decisions on local information only. UMBP through the SGLang KVCache Store Linker replaces that with a direct, layer-wise pipelined HBM-to-DRAM path into a single shared per-node pool, which removes the intermediate copy and stall, deduplicates the replicated KV, and lets placement and eviction be decided globally across DP ranks and instances. (3/3)

Today's Rank

—

Not ranked yet

Combined views

16.4K

2 Sources, first seen 2d ago

50 likes

1 comments

10 saves

3 reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

1 comments

10 saves

3 reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

2 Sources

@SemiAnalysis_HiCache sits between HBM and the external DRAM KV store as a per-rank host-side L2 buffer, so every load and offload takes an extra copy, the L3-to-L2 fetch is not pipelined with compute, and under MLA with TP each rank replicates the same KV across PCIe and host memory while making eviction decisions on local information only. UMBP through the SGLang KVCache Store Linker replaces that with a direct, layer-wise pipelined HBM-to-DRAM path into a single shared per-node pool, which removes the intermediate copy and stall, deduplicates the replicated KV, and lets placement and eviction be decided globally across DP ranks and instances. (3/3)@SemiAnalysis_HiCache sits between HBM and the external DRAM KV store as a per-rank host-side L2 buffer, so every load and offload takes an extra copy, the L3-to-L2 fetch is not pipelined with compute, and under MLA with TP each rank replicates the same KV across PCIe and host memory while making eviction decisions on local information only. UMBP through the SGLang KVCache Store Linker replaces that with a direct, layer-wise pipelined HBM-to-DRAM path into a single shared per-node pool, which removes the intermediate copy and stall, deduplicates the replicated KV, and lets placement and eviction be decided globally across DP ranks and instances. (3/3)

Today's Rank

—

Not ranked yet

Today's Rank

—

Not ranked yet