标题： 高效的大规模语言模型推理：带宽、计算、同步和容量是您所需的一切 显示英文标题

标题： Efficient LLM Inference: Bandwidth, Compute, Synchronization, and Capacity are all you need

摘要： 本文提出了对基于变压器的大语言模型（LLM）推理的极限研究，重点分析了分布式推理系统中由内存带宽、内存容量和同步开销带来的基本性能瓶颈。我们开发了一个与硬件无关的性能模型，抽象掉了实现细节，使得能够分析当前和未来不久的多种硬件技术。我们的分析涵盖了目前在AI加速器如GPU和TPU中使用的HBM3内存技术，以及基于先进HBM4和先进3D堆叠DRAM技术的系统。它还涵盖了基于SRAM的设计和从具有不同芯片数量的分布式集群到晶圆级集成的扩展技术。我们对自回归解码的关键发现是：i) 服务LLM需要每台服务器有数百GB的内存来服务一个模型实例；ii) 高内存带宽对于高用户吞吐量至关重要；iii) 为了实现集体通信而暴露的同步延迟必须在1微秒左右，否则它们会使内存带宽无效；iv) 在以吞吐量每成本或每瓦特衡量的系统级效率方面，基于DRAM的设计具有根本性的优势；v) 硬件设计可以轻松达到2000+用户标记/秒，但要达到10,000+标记/秒将需要更小的模型、更小的上下文或其他形式的算法进步。这项研究为LLM推理的基本性能限制提供了有价值的见解，突显了未来硬件进步的潜在好处，并指导了LLM部署策略的优化。 显示英文摘要

摘要： This paper presents a limit study of transformer-based large language model (LLM) inference, focusing on the fundamental performance bottlenecks imposed by memory bandwidth, memory capacity, and synchronization overhead in distributed inference systems. We develop a hardware-agnostic performance model that abstracts away implementation details, enabling the analysis of a wide range of current and near-future hardware technologies. Our analysis spans from current HBM3 memory technology used in AI accelerators like GPUs and TPUs to systems based on advanced HBM4 and advanced 3D-stacked DRAM technology. It also covers SRAM-based designs and scaling techniques from distributed clusters with varying numbers of chips to wafer-scale integration. Our key findings for auto-regressive decoding are: i) serving LLMs requires 100s of GB per server to serve a model instance; ii) high memory bandwidth is critical for high per-user throughput; iii) exposed synchronization latencies to achieve collective communication must be around 1us else they make the memory bandwidth ineffective; iv) DRAM-based designs have a fundamental advantage in terms of system-level efficiency as measured in throughput per cost or watt; and v) hardware designs can easily reach 2000+ user token/sec but getting to 10,000+ tokens/sec will need smaller models, smaller context, or other forms of algorithmic advances. This study provides valuable insights into the fundamental performance limits of LLM inference, highlighting the potential benefits of future hardware advancements and guiding the optimization of LLM deployment strategies.