Abstract
Transformer-based large language models (LLMs) have demonstrated remarkable potential across a wide range of practical applications. However, long-context inference remains a significant challenge due to the substantial memory requirements of the key-value (KV) cache, which can scale to several gigabytes as sequence length and batch size increase. In this paper, we present \textbf{PackKV}, a generic and efficient KV cache management framework optimized for long-context generation. %, which synergistically supports both latency-critical and throughput-critical inference scenarios. PackKV introduces novel lossy compression techniques specifically tailored to the characteristics of KV cache data, featuring a careful co-design of compression algorithms and system architecture. Our approach is compatible with the dynamically growing nature of the KV cache while preserving high computational efficiency. Experimental results show that, under the same and minimum accuracy drop as state-of-the-art quantization methods, PackKV achieves, on average, \textbf{153.2}\% higher memory reduction rate for the K cache and \textbf{179.6}\% for the V cache. Furthermore, PackKV delivers extremely high execution throughput, effectively eliminating decompression overhead and accelerating the matrix-vector multiplication operation. Specifically, PackKV achieves an average throughput improvement of \textbf{75.7}\% for K and \textbf{171.7}\% for V across A100 and RTX Pro 6000 GPUs, compared to cuBLAS matrix-vector multiplication kernels, while demanding less GPU memory bandwidth. Code available on this https URL
Abstract (translated)
翻译： 基于Transformer的大型语言模型（LLMs）已经在广泛的实用应用中展示了非凡的潜力。然而，长上下文推理仍然是一个重大挑战，因为关键值（KV）缓存的记忆需求随着序列长度和批量大小的增长而显著增加，可能达到数GB之多。在本文中，我们提出了\textbf{PackKV}，这是一个专为长上下文生成优化的通用且高效的KV缓存管理框架。%该方法同时支持延迟敏感型和吞吐量敏感型推理场景。 PackKV引入了专门针对KV缓存数据特性的新颖有损压缩技术，并巧妙地将压缩算法与系统架构相结合设计。我们的方法可以与动态增长性质的KV缓存兼容，同时保持高效的计算效率。实验结果表明，在相同的或最小化的精度下降情况下，与最先进的量化方法相比，PackKV在K缓存中平均实现了\textbf{153.2%}更高的内存减少率，在V缓存中则为\textbf{179.6%}。 此外，PackKV能够实现极高的执行吞吐量，有效消除解压缩开销并加速矩阵向量乘法操作。具体来说，与cuBLAS矩阵向量乘核相比，PackKV在A100和RTX Pro 6000 GPU上分别实现了K平均\textbf{75.7%}和V平均\textbf{171.7%}的吞吐量改进，并且所需的GPU内存带宽更少。 代码可在以下网址获取：[请在此处插入实际链接]
URL
https://arxiv.org/abs/2512.24449
