Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

Researchers developed py-kvcache to analyze the performance of storing large language model key-value states on NVMe SSDs instead of GPU memory. The study finds that while external caching aims to improve efficiency for long-context requests, recomputing data on fast GPUs is often more efficient for short prefixes.