Nvidia pushes AI inference context out to NVMe SSDs

Nvidia has introduced the Inference Context Memory Storage Platform (ICMSP) to address growing KV cache capacity limits in AI inference by standardizing the offload of inference context to NVMe SSDs. This platform, announced at CES 2026, extends GPU KV cache into NVMe-based storage, supported by various storage partners. ICMSP aims to mitigate latency issues caused by the recomputation of evicted context data in large language model inference, enhancing responsiveness and efficiency in AI workloads.