标题： 探索循环大语言模型中的激活稀疏性以实现节能神经形态计算 显示英文标题

标题： Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing

摘要： 大型语言模型（LLMs）的兴起彻底改变了深度学习领域。 然而，希望将LLMs部署在边缘设备上带来了能效和延迟的挑战。 循环LLM（R-LLM）架构已被证明在缓解自注意力的二次复杂性方面有效，使其成为在边缘神经形态处理器上计算的潜在范式。 在本工作中，我们提出了一种低成本、无需训练的算法，以稀疏化R-LLMs的激活，从而在神经形态硬件上提高能效。 我们的方法利用了这些模型的固有结构，使它们非常适合能效受限的环境。 尽管该方法主要针对R-LLMs设计，但可以推广到其他LLM架构，如变换器，如在OPT模型上的演示所示，实现了相当的稀疏性和能效提升。 实证研究表明，我们的方法显著降低了计算需求，同时在多个零样本学习基准上保持了具有竞争力的准确性。 此外，与SENECA神经形态处理器的硬件模拟表明，能效节省和延迟改进显著。 这些结果为低功耗、实时神经形态部署LLMs铺平了道路，并展示了使用激活稀疏性进行无需训练的片上适应的可行性。 显示英文摘要

摘要： The recent rise of Large Language Models (LLMs) has revolutionized the deep learning field. However, the desire to deploy LLMs on edge devices introduces energy efficiency and latency challenges. Recurrent LLM (R-LLM) architectures have proven effective in mitigating the quadratic complexity of self-attention, making them a potential paradigm for computing on-edge neuromorphic processors. In this work, we propose a low-cost, training-free algorithm to sparsify R-LLMs' activations to enhance energy efficiency on neuromorphic hardware. Our approach capitalizes on the inherent structure of these models, rendering them well-suited for energy-constrained environments. Although primarily designed for R-LLMs, this method can be generalized to other LLM architectures, such as transformers, as demonstrated on the OPT model, achieving comparable sparsity and efficiency improvements. Empirical studies illustrate that our method significantly reduces computational demands while maintaining competitive accuracy across multiple zero-shot learning benchmarks. Additionally, hardware simulations with the SENECA neuromorphic processor underscore notable energy savings and latency improvements. These results pave the way for low-power, real-time neuromorphic deployment of LLMs and demonstrate the feasibility of training-free on-chip adaptation using activation sparsity.