标题： 大型语言模型训练和推理的系统性能与成本建模 显示英文标题

标题： System-performance and cost modeling of Large Language Model training and inference

摘要： 基于Transformer架构的大语言模型（LLMs）由于其卓越的可扩展性和适应性，已彻底改变了人工智能、科学和工程等多个领域。 然而，LLM规模和复杂性的指数级增长已经超过了计算能力、内存带宽、网络性能和成本效率的进步，这对分布式系统中的可扩展性提出了重大挑战。 为了解决这些限制，文献中提出了替代模型架构、优化策略、通信感知的网络拓扑结构以及新颖的系统设计方法。 本文介绍了一种用于LLM训练和推理的性能-成本建模方法，该方法结合了最先进的计算技术与内存优化技术以及最新的通信技术。 在分析性能模型的基础上，我们的方法结合了最近的创新，如闪存注意力技术（flash attention technique）和专家混合模型（mixture of experts models），以解决内存带宽和计算瓶颈问题。 它还考虑了不同网络拓扑结构及其特定拓扑的通信算法，采用5D并行性。 该框架还集成了一个芯片组成本模型。 所提出的建模方法提供了有价值的见解，以指导未来的计算系统设计，并促进了硬件-软件协同开发，特别是由于其能够分析各种系统架构配置的性能-成本权衡。 显示英文摘要

摘要： Large language models (LLMs), based on transformer architectures, have revolutionized numerous domains within artificial intelligence, science, and engineering due to their exceptional scalability and adaptability. However, the exponential growth in LLM size and complexity has outpaced advancements in compute capacity, memory bandwidth, network performance, and cost efficiency, posing significant challenges to their scalability on distributed systems. To address these limitations, alternative model architectures, optimization strategies, communication-aware network topologies, and novel system design approaches have been proposed in literature. This paper introduces a performance-cost modeling methodology for LLM training and inference that integrates state-of-the-art compute techniques with memory optimizations, and latest communication techniques. Building on an analytical performance model, our approach incorporates recent innovations such as the flash attention technique and mixture of experts models to address the memory bandwidth and compute bottlenecks. It also considers the impact of different network topologies and topology-specific communication algorithms with 5D parallellism. The framework also integrates a chiplet cost model. The proposed modeling methodology provides valuable insights to guide future compute system design and facilitates hardware-software co-development, in particular due to its ability to analyze performance-cost trade-offs for various system architectural configurations.