标题： 面向构建私有大语言模型：在苹果硅芯片上探索专家并行的多节点混合模型 显示英文标题

标题： Towards Building Private LLMs: Exploring Multi-Node Expert Parallelism on Apple Silicon for Mixture-of-Experts Large Language Model

摘要： 大型语言模型（LLMs）通过OpenAI的ChatGPT、Meta的Llama和Databricks的DBRX等显著进展，彻底改变了人工智能（AI）。 本文解决了在构建用于个人或小团体服务的私有LLM系统时遇到的成本和可扩展性挑战，正如Apple Intelligence所追求的那样。 建立了一个配备Apple M2 Ultra芯片的Mac Studio集群，作为托管和加速具有专家混合（MoE）架构的预训练DBRX模型的成本效益解决方案。 我们的性能分析显示，在两到四个机器节点上并行执行模型的专家显著减少了推理时间。 我们发现专家的计算时间与交换其输出的通信时间相当，强调了网络延迟而非带宽的重要性。 我们还观察到由于Apple软件堆栈的内存管理逻辑而产生的显著管理开销。 基于这些发现，我们开发了优化方案以消除内存管理开销。 结果表明，Mac Studio集群比最先进的配备NVIDIA H100 GPU的AI超级计算机成本效率高1.15倍。 此外，我们构建了一个性能模型，用于估计不同配置下的系统性能，该模型为设计私有LLM系统提供了有价值的见解。 显示英文摘要

摘要： Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI) with significant advancements such as OpenAI's ChatGPT, Meta's Llama, and Databricks' DBRX. This paper addresses the cost and scalability challenges encountered when constructing private LLM systems for personal or small group services, as aimed by Apple Intelligence. A Mac Studio cluster with Apple's M2 Ultra chips is established as a cost-efficient solution to host and accelerate the pretrained DBRX model with the Mixture-of-Experts (MoE) architecture. Our performance analysis reveal that parallel execution of the model's experts across two to four machine nodes significantly reduces inference time. We find that computation time for the experts is comparable to the communication time for exchanging their outputs, emphasizing the importance of network latency over bandwidth. We also observe significant management overhead due to Apple software stack's memory management logic. Based on these findings, we develop optimization schemes to eliminate the memory management overhead. As a result, the Mac Studio cluster is 1.15 times more cost-efficient than the state-of-the-art AI supercomputer with NVIDIA H100 GPUs. In addition, we construct a performance model to estimate system performance under varying configurations, and the model provides valuable insights for designing private LLM systems.