Title: Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs Show Chinese title

Title: 针对 Arm CPU 上的 LLM 推理的高度优化内核和细粒度代码本

Abstract: Large language models (LLMs) have transformed the way we think about language understanding and generation, enthralling both researchers and developers. However, deploying LLMs for inference has been a significant challenge due to their unprecedented size and resource requirements. While quantizing model weights to sub-byte precision has emerged as a promising solution to ease memory pressure, the group quantization formats commonly used for LLM quantization have significant compute overheads and a resource-intensive dequantization process. As a result, a higher proportion of compute instructions do not perform multiplies, i.e., real work, rendering them unsuitable for meeting the required latency requirements for LLMs deployed on commodity CPUs. In this work, we propose a set of highly optimized kernels to accelerate LLM inference and unleash the full potential of CPUs, particularly Arm CPUs. These kernels amortize the cost of loading the operands and the cost of weight unpacking across multiple output rows. This, along with the introduction of an optimized interleaved group data layout for weights and decompression path optimizations to reduce unnecessary operations and dequantization overhead while maximizing the use of vector and matrix multiply operations, significantly improves the efficiency of MAC operations. Furthermore, we present a groupwise non-uniform codebook-based quantization method for ultra-low-precision quantization of LLMs to better match non-uniform patterns in their weight distributions, demonstrating better throughput during token generation while ensuring better quality than the state-of-the-art. Applying these improvements to 4-bit LLMs results in a 3-3.2x improvement in prompt processing and a 2x improvement in autoregressive decoding on Arm CPUs, compared to LLaMA.cpp-based solution. The optimized kernels are available at https://github.com/ggerganov/llama.cpp. Show Chinese abstract

Abstract: 大型语言模型（LLMs）已经改变了我们对语言理解和生成的看法，吸引了研究人员和开发者的极大兴趣。然而，由于其前所未有的规模和资源需求，将LLMs用于推理部署一直是一项重大挑战。 尽管将模型权重量化到亚字节精度已成为缓解内存压力的一种有前景的解决方案，但LLMs量化中常用的组量化格式具有显著的计算开销和资源密集型的反量化过程。 因此，更多的计算指令不执行乘法操作，即实际工作，这使得它们不适合满足商品CPU上部署的LLMs所需的延迟要求。 在这项工作中，我们提出了一组高度优化的内核，以加速LLMs推理并释放CPU，特别是Arm CPU的全部潜力。 这些内核将操作数加载的成本和权重解包的成本分摊到多个输出行上。 此外，通过引入一种优化的交错组数据布局来减少不必要的操作和反量化开销，并最大化向量和矩阵乘法操作的使用，显著提高了MAC操作的效率。 此外，我们提出了一个基于组非均匀码本的量化方法，用于LLMs的超低精度量化，以更好地匹配其权重分布中的非均匀模式，在确保优于最先进的质量的同时，在标记生成过程中表现出更好的吞吐量。 将这些改进应用于4位LLMs，在Arm CPU上相比基于LLaMA.cpp的解决方案，提示处理速度提高了3-3.2倍，自回归解码速度提高了2倍。 优化后的内核可在https://github.com/ggerganov/llama.cpp获取。