标题： 表示迁移：将标记压缩与FlashAttention统一起来 显示英文标题

标题： Representation Shift: Unifying Token Compression with FlashAttention

摘要： Transformer在视觉、语言和视频领域展示了显著的成功。 然而，任务复杂性的增加导致了更大的模型和更多的标记，增加了自注意力的二次成本以及GPU内存访问的开销。 为了减少自注意力的计算成本，先前的工作提出了标记压缩技术，以删除冗余或信息量较少的标记。 同时，像FlashAttention这样的融合注意力内核被开发出来，通过避免注意力图的构建及其相关的HBM I/O来减轻内存开销。 然而，这使得它与大多数无需训练的标记压缩方法不兼容，这些方法依赖于注意力图来确定标记的重要性。 在此，我们提出了Representation Shift，这是一种无需训练、与模型无关的度量，用于衡量每个标记表示的变化程度。 这可以无缝地将标记压缩与FlashAttention结合，而无需注意力图或重新训练。 我们的方法进一步推广到CNN和状态空间模型。 大量实验表明，Representation Shift使与FlashAttention兼容的有效标记压缩成为可能，在视频文本检索和视频问答中分别实现了高达5.5%和4.4%的速度提升。 代码可在https://github.com/mlvlab/Representation-Shift获取。 显示英文摘要

摘要： Transformers have demonstrated remarkable success across vision, language, and video. Yet, increasing task complexity has led to larger models and more tokens, raising the quadratic cost of self-attention and the overhead of GPU memory access. To reduce the computation cost of self-attention, prior work has proposed token compression techniques that drop redundant or less informative tokens. Meanwhile, fused attention kernels such as FlashAttention have been developed to alleviate memory overhead by avoiding attention map construction and its associated I/O to HBM. This, however, makes it incompatible with most training-free token compression methods, which rely on attention maps to determine token importance. Here, we propose Representation Shift, a training-free, model-agnostic metric that measures the degree of change in each token's representation. This seamlessly integrates token compression with FlashAttention, without attention maps or retraining. Our method further generalizes beyond Transformers to CNNs and state space models. Extensive experiments show that Representation Shift enables effective token compression compatible with FlashAttention, yielding significant speedups of up to 5.5% and 4.4% in video-text retrieval and video QA, respectively. Code is available at https://github.com/mlvlab/Representation-Shift.