标题： SiPipe：填补CPU-GPU利用率差距以实现高效流水线并行大语言模型推理 显示英文标题

标题： SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference

摘要： 随着大型语言模型（LLMs）的推理工作负载规模扩大以满足不断增长的用户需求，管道并行性（PP）已成为多GPU部署中广泛采用的策略，特别是在跨节点设置中，以提高键值（KV）缓存容量和推理吞吐量。 然而，PP由于三种类型的执行气泡——负载不平衡、阶段内和阶段间——导致固有的低效性，限制了管道的饱和度。 我们提出了SiPipe，一种异构管道设计，通过利用未充分利用的CPU资源来卸载辅助计算和通信，从而提高吞吐量。 SiPipe结合了三种关键技术——CPU采样、令牌安全执行模型和结构感知传输——以减轻管道气泡并提高执行效率。 在多种LLMs上，与相同PP配置下的最先进vLLM相比，SiPipe实现了高达2.1倍的更高吞吐量，每令牌延迟降低了43%，以及高达23%的平均GPU利用率，证明了其在LLMs和部署场景中的通用性。 显示英文摘要

摘要： As inference workloads for large language models (LLMs) scale to meet growing user demand, pipeline parallelism (PP) has become a widely adopted strategy for multi-GPU deployment, particularly in cross-node setups, to improve key-value (KV) cache capacity and inference throughput. However, PP suffers from inherent inefficiencies caused by three types of execution bubbles-load-imbalance, intra-stage, and inter-stage-which limit pipeline saturation. We present SiPipe, a heterogeneous pipeline design that improves throughput by leveraging underutilized CPU resources to offload auxiliary computation and communication. SiPipe incorporates three key techniques-CPU sampling, a token-safe execution model, and structure-aware transmission-to mitigate pipeline bubbles and improve execution efficiency. Across diverse LLMs, SiPipe achieves up to 2.1 times higher throughput, 43% lower per-token latency, and up to 23% higher average GPU utilization compared to the state-of-the-art vLLM under the same PP configuration, demonstrating its generality across LLMs and deployment scenarios.