标题： ELFATT：用于视觉变压器的高效线性快速注意力 显示英文标题

标题： ELFATT: Efficient Linear Fast Attention for Vision Transformers

摘要： 注意力机制是变压器在不同机器学习任务中取得成功的关键。 然而，传统的基于softmax的注意力机制相对于序列长度的二次复杂度成为长序列任务（如视觉任务）应用的主要瓶颈。 尽管已经提出了各种高效的线性注意力机制，但它们需要牺牲性能以实现高效率。 此外，像FlashAttention-1-3这样的内存高效方法仍然具有二次计算复杂度，可以进一步改进。 在本文中，我们提出了一种新颖的高效线性快速注意力（ELFATT）机制，以同时实现低内存输入/输出操作、线性计算复杂度和高性能。 在高分辨率视觉任务中，ELFATT相比传统的基于softmax的注意力机制提供了4-7倍的速度提升，而不会损失性能。 ELFATT兼容FlashAttention。 使用FlashAttention-2加速，ELFATT在高分辨率视觉任务中相比传统的基于softmax的注意力机制仍提供2-3倍的速度提升，而不会损失性能。 即使在边缘GPU上，ELFATT在各种功率模式（从5W到60W）下与最先进的注意力机制相比仍能提供1.6倍至2.0倍的速度提升。 此外，ELFATT可以直接用于增强和加速扩散任务，而无需训练。 显示英文摘要

摘要： The attention mechanism is the key to the success of transformers in different machine learning tasks. However, the quadratic complexity with respect to the sequence length of the vanilla softmax-based attention mechanism becomes the major bottleneck for the application of long sequence tasks, such as vision tasks. Although various efficient linear attention mechanisms have been proposed, they need to sacrifice performance to achieve high efficiency. What's more, memory-efficient methods, such as FlashAttention-1-3, still have quadratic computation complexity which can be further improved. In this paper, we propose a novel efficient linear fast attention (ELFATT) mechanism to achieve low memory input/output operations, linear computational complexity, and high performance at the same time. ELFATT offers 4-7x speedups over the vanilla softmax-based attention mechanism in high-resolution vision tasks without losing performance. ELFATT is FlashAttention friendly. Using FlashAttention-2 acceleration, ELFATT still offers 2-3x speedups over the vanilla softmax-based attention mechanism on high-resolution vision tasks without losing performance. Even on edge GPUs, ELFATT still offers 1.6x to 2.0x speedups compared to state-of-the-art attention mechanisms in various power modes from 5W to 60W. Furthermore, ELFATT can be used to enhance and accelerate diffusion tasks directly without training.