标题： 具有优化数据流的边缘大语言模型推理混合流水线阵列加速器 显示英文标题

标题： Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference

摘要： 边缘推理为大型语言模型（LLM）提供了安全、低延迟和成本效益高的推理解决方案。 我们强调，边缘加速器应在内存密集的解码阶段实现高面积效率并最小化外部内存访问（EMA），同时在计算密集的预填充阶段保持高能效。 本文提出了一种具有混合脉动阵列（HSA）架构的边缘LLM推理加速器，该架构优化了两个阶段的推理效率。 为了进一步减少EMA，我们采用了MXINT4权重量化，并提出了一种针对HSA优化的数据流，确保可忽略的反量化开销，并在边缘DRAM带宽限制下实现100%的硬件利用率，同时仅造成最小的精度损失。 对于非线性操作，我们集成了优化的均方根归一化（RMSNorm）和旋转位置嵌入（RoPE）单元，在减少其延迟、面积和内存访问开销的同时，实现了在我们的加速器上的端到端推理。 我们的解决方案在运行一个1.3B LLM的长输入/长输出场景时，达到了247/117（token/s/mm2），相比现有方法提供了超过2.45倍/13.5倍的提升，同时在标记生成过程中保持了卓越的能效。 显示英文摘要

摘要： Edge inference for large language models (LLM) offers secure, low-latency, and cost-effective inference solutions. We emphasize that an edge accelerator should achieve high area efficiency and minimize external memory access (EMA) during the memory-bound decode stage, while maintaining high energy efficiency during the compute intensive prefill stage. This paper proposes an edge LLM inference accelerator featuring a hybrid systolic array (HSA) architecture that optimizes inference efficiency in both stages. To further reduce EMA, we adopt MXINT4 weight quantization and propose an optimized dataflow tailored for HSA, ensuring negligible dequantization overhead and achieving 100% hardware utilization with minimal accuracy loss under edge DRAM bandwidth constraints. For non-linear operations, we incorporate optimized root mean square normalization (RMSNorm) and rotary position embedding (RoPE) units, reducing their latency, area, and memory access overhead while enabling end-to-end inference on our accelerator. Our solution achieves 247/117 (token/s/mm2) while running a 1.3B LLM on long-input/long-output scenarios, providing >2.45x/13.5x improvement over existing approaches, while maintaining superior energy efficiency in token generation.