Title: SQuat: Subspace-orthogonal KV Cache Quantization Show Chinese title

Title: SQuat：子空间正交KV缓存量化

Abstract: The key-value (KV) cache accelerates LLMs decoding by storing KV tensors from previously generated tokens. It reduces redundant computation at the cost of increased memory usage. To mitigate this overhead, existing approaches compress KV tensors into lower-bit representations; however, quantization errors can accumulate as more tokens are generated, potentially resulting in undesired outputs. In this paper, we introduce SQuat (Subspace-orthogonal KV cache quantization). It first constructs a subspace spanned by query tensors to capture the most critical task-related information. During key tensor quantization, it enforces that the difference between the (de)quantized and original keys remains orthogonal to this subspace, minimizing the impact of quantization errors on the attention mechanism's outputs. SQuat requires no model fine-tuning, no additional calibration dataset for offline learning, and is grounded in a theoretical framework we develop. Through numerical experiments, we show that our method reduces peak memory by 2.17 to 2.82, improves throughput by 2.45 to 3.60, and achieves more favorable benchmark scores than existing KV cache quantization algorithms. Show Chinese abstract

Abstract: 键值（KV）缓存通过存储之前生成标记的KV张量来加速LLMs解码。 它以增加内存使用为代价减少了冗余计算。 为了减轻这种开销，现有方法将KV张量压缩为低比特表示；然而，随着更多标记的生成，量化误差可能会累积，可能导致不期望的输出。 在本文中，我们引入了SQuat（子空间正交KV缓存量化）。 它首先构建由查询张量张成的子空间，以捕捉最关键的任务相关信息。 在键张量量化过程中，它强制（反量化）后的键与原始键之间的差异保持与此子空间正交，从而最小化量化误差对注意力机制输出的影响。 SQuat不需要模型微调，不需要额外的校准数据集进行离线学习，并基于我们开发的理论框架。 通过数值实验，我们证明我们的方法将峰值内存减少了2.17到2.82，吞吐量提高了2.45到3.60，并且比现有的KV缓存量化算法取得了更有利的基准分数。