标题： 面向脉冲变压器中的相对位置编码 显示英文标题

标题： Toward Relative Positional Encoding in Spiking Transformers

摘要： 脉冲神经网络（SNNs）是受生物启发的网络，它们通过离散的脉冲模仿大脑中神经元之间的通信，由于其能效和时间处理能力，在各种任务中具有巨大的潜力。 具有自注意力机制的SNNs（脉冲Transformer）最近在各种任务中表现出显著的进步，并且受到传统Transformer的启发，一些研究已经证明，脉冲绝对位置编码有助于捕捉输入数据的序列关系，增强了脉冲Transformer在序列建模和图像分类等任务中的能力。 然而，如何将相对位置信息融入SNNs仍然是一个挑战。 在本文中，我们引入了几种策略来近似脉冲Transformer中的相对位置编码（RPE），同时保持脉冲的二进制特性。 首先，我们正式证明了使用格雷码对相对距离进行编码可以确保当其十进制值相差为2的幂时，位置索引的二进制表示保持恒定的汉明距离，并基于此特性提出了基于格雷码的位置编码（Gray-PE）。 此外，我们提出了一种称为Log-PE的另一种RPE方法，该方法直接将相对距离矩阵的对数形式结合到脉冲注意力图中。 此外，我们将我们的RPE方法扩展为二维形式，使其适用于处理图像块。 我们在各种任务上评估了我们的RPE方法，包括时间序列预测、文本分类和基于块的图像分类，实验结果表明，在许多架构中引入我们的RPE方法都能带来令人满意的性能提升。 显示英文摘要

摘要： Spiking neural networks (SNNs) are bio-inspired networks that mimic how neurons in the brain communicate through discrete spikes, which have great potential in various tasks due to their energy efficiency and temporal processing capabilities. SNNs with self-attention mechanisms (spiking Transformers) have recently shown great advancements in various tasks, and inspired by traditional Transformers, several studies have demonstrated that spiking absolute positional encoding can help capture sequential relationships for input data, enhancing the capabilities of spiking Transformers for tasks such as sequential modeling and image classification. However, how to incorporate relative positional information into SNNs remains a challenge. In this paper, we introduce several strategies to approximate relative positional encoding (RPE) in spiking Transformers while preserving the binary nature of spikes. Firstly, we formally prove that encoding relative distances with Gray Code ensures that the binary representations of positional indices maintain a constant Hamming distance whenever their decimal values differ by a power of two, and we propose Gray-PE based on this property. In addition, we propose another RPE method called Log-PE, which combines the logarithmic form of the relative distance matrix directly into the spiking attention map. Furthermore, we extend our RPE methods to a two-dimensional form, making them suitable for processing image patches. We evaluate our RPE methods on various tasks, including time series forecasting, text classification, and patch-based image classification, and the experimental results demonstrate a satisfying performance gain by incorporating our RPE methods across many architectures.