标题： 面向高效键值缓存管理的LLM推理前缀预填充策略 显示英文标题

标题： Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference

摘要： 随着具有扩展上下文窗口的大规模语言模型（LLMs）的广泛应用，高效的关键值缓存（KVC）管理成为优化推理性能的必要条件。检索增强生成（RAG）和代理等推理工作负载表现出较高的缓存可重用性，这使得高效的缓存管理对于减少冗余和提高速度至关重要。我们使用公开可用的跟踪数据分析了实际的KVC访问模式，并评估了Redis等商业键值存储以及基于RDMA的系统（CHIME [1] 和Sherman [2]）用于KVC元数据管理的效果。我们的研究展示了针对KVC预填充缺乏专门存储解决方案的情况，强调了为LLM工作负载设计具有优化元数据管理的高效分布式缓存系统的必要性，并为设计可扩展且低延迟的KVC管理系统提供了见解。 显示英文摘要

摘要： The increasing adoption of large language models (LLMs) with extended context windows necessitates efficient Key-Value Cache (KVC) management to optimize inference performance. Inference workloads like Retrieval-Augmented Generation (RAG) and agents exhibit high cache reusability, making efficient caching critical to reducing redundancy and improving speed. We analyze real-world KVC access patterns using publicly available traces and evaluate commercial key-value stores like Redis and state-of-the-art RDMA-based systems (CHIME [1] and Sherman [2]) for KVC metadata management. Our work demonstrates the lack of tailored storage solution for KVC prefilling, underscores the need for an efficient distributed caching system with optimized metadata management for LLM workloads, and provides insights into designing improved KVC management systems for scalable, low-latency inference.