标题： 共享 DIFF Transformer 显示英文标题

标题： Shared DIFF Transformer

摘要： DIFF Transformer通过增强对相关上下文的关注并抑制噪声来改进注意力分配。 它引入了一种差分注意力机制，计算两个独立生成的注意力分布之间的差异，有效减少噪声并促进稀疏注意力模式。 然而，DIFF Transformer中的独立信号生成导致参数冗余和信息利用不充分。 在本工作中，我们提出了共享DIFF Transformer，通过引入一个共享的基础矩阵来建模全局模式，并结合低秩更新以增强任务特定的灵活性，借鉴了差分放大器的思想。 这种设计显著减少了参数冗余，提高了效率，并保留了强大的噪声抑制能力。 实验结果表明，与DIFF Transformer相比，我们的方法在长序列建模、关键信息检索和上下文学习等任务中表现更好。 我们的工作提供了一种新颖且高效的方法来优化差分注意力机制，并推动了鲁棒的Transformer架构的发展。 显示英文摘要

摘要： DIFF Transformer improves attention allocation by enhancing focus on relevant context while suppressing noise. It introduces a differential attention mechanism that calculates the difference between two independently generated attention distributions, effectively reducing noise and promoting sparse attention patterns. However, the independent signal generation in DIFF Transformer results in parameter redundancy and suboptimal utilization of information. In this work, we propose Shared DIFF Transformer, which draws on the idea of a differential amplifier by introducing a shared base matrix to model global patterns and incorporating low-rank updates to enhance task-specific flexibility. This design significantly reduces parameter redundancy, improves efficiency, and retains strong noise suppression capabilities. Experimental results show that, compared to DIFF Transformer, our method achieves better performance in tasks such as long-sequence modeling, key information retrieval, and in-context learning. Our work provides a novel and efficient approach to optimizing differential attention mechanisms and advancing robust Transformer architectures.