Title: PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices Show Chinese title

Title: PIPO：面向消费设备高效推理的流水线卸载技术

Abstract: The high memory and computation demand of large language models (LLMs) makes them challenging to be deployed on consumer devices due to limited GPU memory. Offloading can mitigate the memory constraint but often suffers from low GPU utilization, leading to low inference efficiency. In this work, we propose a novel framework, called pipelined offloading (PIPO), for efficient inference on consumer devices. PIPO designs a fine-grained offloading pipeline, complemented with optimized data transfer and computation, to achieve high concurrency and efficient scheduling for inference. Experimental results show that compared with state-of-the-art baseline, PIPO increases GPU utilization from below 40% to over 90% and achieves up to 3.1$\times$ higher throughput, running on a laptop equipped with a RTX3060 GPU of 6GB memory. Show Chinese abstract

Abstract: 大型语言模型（LLMs）对内存和计算能力的高需求，由于消费者设备上的GPU内存有限，使得它们部署起来颇具挑战性。卸载可以缓解内存限制，但常常面临GPU利用率低的问题，导致推理效率低下。在这项工作中，我们提出了一种名为管道卸载（PIPO）的新框架，用于消费者设备上的高效推理。PIPO设计了一个细粒度的卸载流水线，并辅以优化的数据传输和计算，以实现高并发性和高效的调度来进行推理。实验结果显示，与最先进的基线相比，PIPO将GPU利用率从低于40%提高到超过90%，并在配备6GB内存RTX3060 GPU的笔记本电脑上实现了高达3.1$\times$更高的吞吐量。